JP2013097616A - User operation log recording method and device thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To quickly acquire an acquisition frequency of a state of a GUI to record contents of high reproducibility of the state of the GUI into a log.SOLUTION: A direct access function is used to acquire the present state of a GUI as an initial state and outputs it to a storage device as a log to record it (S31), a listener for detecting a GUI state change event or a window extinction event in windows other than the forefront window by an event detection function is set (S32), the direct access function is used to output and record the state of the GUI in the forefront window to and in the storage device as a log (S34 and S35) if designation of acquisition timing is satisfied before input of an instruction to terminate log recording (S33 and S38), and further, it is examined whether the event has been detected by the event detection function or not, and event information of the event is outputted to the storage device and is recorded (S36 and S37).

Description

  The present invention relates to a technique for recording a user operation log for a graphical user interface.

  In recent years, a number of technologies that utilize user operation logs have been proposed as solutions for information processing systems such as business systems in companies that are realized by software such as application programs (hereinafter referred to as applications) that run on computers. Has been. These records and accumulates user operations in the computer terminal devices (hereinafter referred to as information system terminals) constituting the information processing system, and realizes that operations that violate security are not performed, It is used for applications that achieve business efficiency by analyzing bottlenecks.

  Many of the information processing systems described above employ a graphical user interface (hereinafter, GUI) as a user interface for improving usability. Therefore, it is necessary to record and store user operations on the GUI even in the technique using the user operation log.

  User operations on the GUI are mainly mouse operations and keyboard inputs, and can be detected as events of the operating system (hereinafter referred to as OS) of the information system terminal together with information such as the occurrence time. In addition, regarding the GUI state, it is possible to detect the occurrence of the opening or closing of a window as information about the process name of the application, the occurrence time of the event, etc. as an OS event.

  Many of the techniques using the user operation log record and store events generally included in the OS as a log. For security purposes, when performing simple analysis such as grasping user operations and window activation of the application, it is considered that the contents to be recorded in the log are sufficient. However, in order for the user to know exactly what GUI component in the window he / she did during his work, and how the display in the window changed as a result, he / she needs to record Insufficient information

  In security and business analysis, grasping the situation (context) at the time of user operation is important for improving the accuracy of analysis. For example, as an analysis viewpoint, consider a case where an operation of pressing the initialization button for data displayed in an application window is detected as a violation operation from a log. However, not all such operations are actually illegal operations. A trouble may have occurred, and it may have been necessary to initialize the data, or an operation performed by the administrator as needed. To determine whether or not the operation is actually violated, it is necessary to investigate the display state of the GUI component in the application window before or when the data initialization button is pressed or to grasp the context of the user operation. .

  In this way, in order to accurately grasp the contents and intentions of user operations, not only events related to mouse operations, keyboard input, and application window opening / closing, but also the display status of GUI components in windows are recorded in a log. There is a need to.

[Conventional technology]
There are roughly two methods for recording the display state of the GUI component in the application window as a log.

  The first method is a method of incorporating dedicated software (plug-in) for outputting the display state of the GUI component in the window into an information system terminal or application for acquiring a log. Although this method can reliably acquire information on the display state of GUI parts if a plug-in is applicable, it is not very versatile because some information system terminals and applications cannot be installed in the first place. (See Non-Patent Document 1).

  The second method is a method of recording the screen or moving image of the display device in the information system terminal and identifying the display state in the window. This method has a problem that the result depends on the accuracy of the technology for identifying the display state of the GUI component in the window. However, since the screen and video of the display device can be acquired regardless of the information system terminal or application, It can be said that this is a highly efficient technique (see Non-Patent Document 2).

  In particular, in the former method, GUI accessibility technologies such as UI automation (see non-patent document 3) and Java (registered trademark) Accessibility API (see non-patent document 4) have recently advanced, and .NET, Java (registered trademark), In a Web-based application, it is possible to output the display state of the GUI component in the window in a general purpose to some extent. In addition, standardization of accessibility technology (see Non-Patent Document 5) is progressing with respect to applications having a GUI, and it is assumed that applications to which such GUI accessibility technology can be applied will increase in the future. Thus, in the case of the former method, the versatility is also improving.

  Many of GUI accessibility technologies directly access an application from a log acquisition program and acquire the display state of a GUI component in a window (hereinafter referred to as “direct access function”), and a log acquisition program is a listener (for some event). 2) a function (hereinafter referred to as an “event detection function”) that detects a change in the display state of a GUI component in the window as an event. It is common to have one.

  Hereinafter, the display state of the window displayed on the screen of the display device by the software and the display state of the GUI component in the window are simply referred to as “GUI state”.

  By the way, when acquiring a log of a user operation, an application must be started on an information system terminal and a log acquisition program must be operated in the background. Therefore, the log acquisition program is required not to deteriorate the response of the application to the user operation even if it is operated in parallel with the application. However, when recording not only the operation of the mouse and the input to the keyboard but also the state of the GUI of the application, the information to be recorded becomes enormous. As a result, the processing time of the log acquisition program becomes longer, which may affect the response of the application.

  To solve this problem, for example, the information to be detected and recorded from the application is limited to the information necessary for analysis, and the GUI state acquisition processing time can be shortened by using a method for reducing the GUI state acquisition processing time. A possible countermeasure is to reduce the acquisition frequency. Alternatively, there is a method of detecting and recording only the difference changed from the previous GUI state using the event detection function. These methods are effective in terms of improving the response performance of the application to the user operation, but on the other hand, the quality is lowered in that the GUI state of the application to the user operation is reproduced from the log. .

  Further, if information to be detected and recorded from an application is limited, it is impossible to reproduce the operation status for information that is not of interest. Information that is not recorded in the log cannot be analyzed later. Therefore, when this method is adopted, it is necessary to clarify information of interest in advance. However, in log analysis, the information of interest often changes during the analysis process, and it is difficult to clarify the information of interest in advance.

  If the acquisition frequency of the GUI state using the direct access function is lowered, it is impossible to reproduce a detailed transition of the GUI state from a certain timing at which the state is acquired to a next timing at which the state is acquired.

  Further, when only the difference changed from the previous GUI state is detected by using the event detection function, the transition of the GUI state can surely be reproduced in detail. However, on the other hand, depending on the type of application or information, there may be a case where it cannot be detected as an event. For example, in general, when a large number of events occur due to a change in GUI and an overflow occurs in the event queue, these events are not notified to the log acquisition program and are discarded. Therefore, in this method, there is a problem that the probability that the state of the GUI of the application and the state of the GUI recorded in the log do not match increases as the time for acquiring the log increases, and the degree of mismatch increases. is there.

  As described above, in the log acquisition program, the GUI state is acquired at high speed according to the response performance of the information system terminal to the user operation or the response performance of the log recording target software, and the content of the GUI state is highly reproducible. Needs to be solved.

  The present invention has the following features in order to solve the problem of acquiring the GUI state at high speed and recording highly reproducible contents of the GUI state in a log.

As the first invention of the present invention,
Log of user operation for GUI provided by the log recording target software operating on an arbitrary computer via the input device and the display device of the arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path The log display target software displays a GUI display state as a GUI on the screen of the display device and a GUI state represented by the display state of the GUI component in the window. A method of recording in a storage device of another computer connected to a computer through an arbitrary communication path,
The arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path,
Registers in the storage device the conditions for specifying the GUI component or its attribute information to be excluded from the log acquisition target and the timing for acquiring the GUI status of the log recording target software using the direct access function. A first step to:
A second step of acquiring a current GUI state as an initial state using a direct access function in response to a log recording start instruction input from an input device, and outputting and recording the log to a storage device;
And a third step of setting a listener for detecting a GUI state change event other than the frontmost window or a window extinction event by an event detection function,
After the third step, the process is repeated until the log recording end instruction is input from the input device.
A fourth step of outputting and recording the GUI state of the foremost window as a log to the storage device in accordance with the designation of the timing for acquiring the GUI state of the log recording target software using the direct access function;
A fifth step of checking whether a change event of the GUI state other than the frontmost window or a window extinction event is detected by the event detection function, and outputting the event information to the storage device as a log and recording it. Features.

As a second invention of the present invention,
In the fourth step in the first invention,
Along with the time stamp information, the direct access function is used to obtain the GUI state of the frontmost window directly from the application, the window or GUI component is a node, the inclusion relationship is expressed as a parent-child relationship between the nodes, and each node Is output in an expression format having attribute information, and excludes from the output nodes or attribute information that match a GUI component to be excluded from an input target of log input or a condition for specifying the attribute information.

As a third invention of the present invention,
In the fourth step in the first invention,
In the specification of the timing for acquiring the application GUI status using the direct access function, the log output / recording frequency can be changed by specifying the time, the interval, or the user operation.

As a fourth invention of the present invention,
In the fifth step of the first invention,
Time stamp information at the time of event detection, change of attribute information value other than the frontmost window of the event detected by the event detection function, generation of GUI parts other than the frontmost window, or GUI other than the window or frontmost window Along with information on the event type that indicates which of the parts disappeared,
If it is a change event of the value of attribute information other than the frontmost window, the information specifying the node in the tree structure representation of the GUI state corresponding to the change target, the attribute name to be changed and the value after the change,
Further, if it is a generation event of a GUI component other than the frontmost window, information for specifying a node in the tree structure representation of the GUI state corresponding to the generation target, attribute information for the generation target, and information for specifying the parent node,
Further, if the event is a change event of disappearance of a GUI component other than the window or the frontmost window, information specifying a node in the tree structure representation of the GUI state corresponding to the disappearance target is output.

As a fifth invention of the present invention,
Log of user operation for GUI provided by the log recording target software operating on an arbitrary computer via the input device and the display device of the arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path The log display target software displays a GUI display state as a GUI on the screen of the display device and a GUI state represented by the display state of the GUI component in the window. A device for recording in a storage device of another computer connected to the computer through an arbitrary communication path,
The next device provided on the arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path, that is, an input device;
A storage device;
A direct access function unit for directly accessing the log recording target software and obtaining a GUI state or a GUI state of only the frontmost window;
An event detection function unit for setting a listener that detects a GUI state change event other than the frontmost window or a window disappearance event, and detects these events that occur in response to a GUI state change;
When the designation of the GUI component to be excluded from the log acquisition target or the attribute information thereof and the specification of the timing for acquiring the GUI status of the log recording target software using the direct access function unit are input from the input device Is registered in the storage device, and when an instruction to start log recording is input from the input device, the current GUI state is acquired as an initial state using the direct access function unit, and is output to the storage device as a log for recording. Then, a listener for detecting a GUI state change event other than the frontmost window or a window disappearance event is set using the event detection function unit, and thereafter, an instruction to end log recording is input from the input device. According to the specification of the timing for acquiring the GUI status of the log recording target software using the direct access function unit The GUI status of the frontmost window is output to the storage device as a log and recorded. Further, the event detection function unit checks whether a change event of the GUI status other than the frontmost window or a window disappearance event is detected. A log acquisition control unit that outputs and records information as a log to a storage device.

  According to the present invention, when the GUI state is recorded as a user operation log by using GUI accessibility technology or a plug-in, the software of the user operation according to the performance of the information system terminal or the log recording target software is recorded. It is possible to record information that can reproduce the GUI state in detail without impairing the response performance.

It is a block diagram which shows an example of embodiment of the apparatus which implements the method of recording the operation log of this invention. It is a processing flow of the initialization procedure of the direct access function and event detection function in the present invention. It is a process flow of the log recording procedure of the state of GUI in this invention. It is explanatory drawing which shows the example which expressed the state of the GUI in an application, and this using the tree structure. It is explanatory drawing which shows the example of the expression format of the attribute information which the node in a tree structure hold | maintains. It is explanatory drawing which shows the example of the conditions which designate the GUI component excluded from the acquisition target of a log, or its attribute information. It is explanatory drawing which shows the output content of the log according to the kind of event at the time of a change event detection. FIG. 5 is an explanatory diagram showing a tree structure representation acquired with the GUI state of FIG. 4 as an initial state. It is explanatory drawing which shows the example of the state of GUI after user operation. It is explanatory drawing which shows the tree structure representation which acquired the state of GUI of FIG. It is explanatory drawing which shows the recording content of the change event of the state of GUI from FIG. 4 to FIG. It is explanatory drawing which shows another example of the state of GUI after user operation. It is explanatory drawing which shows the tree structure expression which acquired the state of GUI of FIG. It is explanatory drawing which shows another example of the state of GUI after user operation. It is explanatory drawing which shows the tree structure representation which acquired the state of GUI of FIG. It is explanatory drawing which shows the recording content of the change event of the state of GUI from FIG. 12 to FIG.

  FIG. 1 shows an example of an embodiment of an apparatus for carrying out a user operation log recording method of the present invention, here an example realized on a well-known computer. In the figure, 1 is an input device, 2 is a storage device 3 is a display device, and 4 is a computer.

  The input device 1 includes a keyboard, a mouse, and the like. The user uses the computer 4 to specify predetermined information, here, a GUI component to be excluded from log acquisition targets or its attribute information, and an application (log recording) using a direct access function. This is for specifying the timing for acquiring the GUI state of the target software) and inputting the log recording start and log recording end instructions. In addition, designation of the type of user operation originally recorded in the log is also input. This is, for example, a mouse operation, a click operation, a double-click operation, a drag operation, the type of a pressed mouse button, etc., and an input to the keyboard, the contents of the pressed key, key down, key up, etc. Etc. The present invention focuses on a method for recording a GUI state in a log when a user operates. In order to simplify the explanation, the recording method of the type of user operation will be omitted as it is the same as a known technique.

  The storage device 2 stores the above-described predetermined information as an initial set value and records a log acquired and output by the procedure of FIGS. 2 and 3 performed by the computer 4. The display device 3 is composed of a liquid crystal display, a CRT, or the like, and displays an application screen and also displays a screen in the procedure of FIGS. 2 and 3 executed by the computer 4.

  The computer 4 includes a direct access function unit 5 and an event detection function unit 6 realized by the above-described known GUI accessibility technology. Based on the information input from the input device 1, the computer 4 follows the procedure shown in FIGS. A user operation log is acquired using the direct access function unit 5 and the event detection function unit 6, and is output and recorded in the storage device 2. At this time, the log acquisition control unit 7 is configured.

  The direct access function unit 5 directly accesses the application to construct a tree structure representation of the GUI state in the steps S31 and S35 of the procedure of FIG. Get only GUI status. In addition, the event detection function unit 6 performs a GUI state change event or window disappearance event other than the foremost window in step S32 of the procedure of FIG. Change the attribute information value, create a GUI part other than the frontmost window, or set a listener to detect the disappearance event of the GUI part other than the window or frontmost window, and change the GUI state in step S37 Detect these events that occur along with.

  The log acquisition control unit 7 receives from the input device 1 conditions for designating GUI components to be excluded from log acquisition targets or attribute information thereof, and designation of timing for acquiring the application GUI status using the direct access function. Then, the procedure of FIG. 2 is performed, and the direct access function unit 5 and the event detection function unit 6 are initialized.

  The initial setting of the direct access function unit 5 in step S21 in the procedure of FIG. 2 specifically includes conditions for specifying a GUI component to be excluded from acquisition targets of logs input from the input device 1 or attribute information thereof, and direct access. The designation of the timing for obtaining the GUI state of the application using the function unit 5 is registered in the storage device 2.

  Hereinafter, first, conditions for designating GUI parts to be excluded from log acquisition targets or attribute information thereof will be described.

  As described above, one of the objects of the present invention is to record in a log so that the state of the GUI when the user performs an operation can be reproduced. The GUI state is formally expressed by a set of application display windows. The displayed window is represented by a tree structure in which the window and GUI parts displayed in the window are nodes. The parent-child relationship between nodes corresponds to the display inclusion relationship between windows and GUI components. For example, when there is a GUI part of a button in the window, a parent-child relationship exists between a node corresponding to the window and a node corresponding to the button.

  A window or GUI component generally holds attribute information. For example, the display position, size, name, GUI part type (label, text box, combo box, etc.), process ID, and the like can be given. In addition, if it is a window, information such as whether or not the user can operate the window and internal GUI parts, information on whether it is displayed in the foreground, and GUI parts such as labels are displayed. If it is a GUI part such as a text string, its font, font size, color, or a GUI part such as a text box, its setting value is added. Each node holds the value of such attribute information in the corresponding GUI component. It should be noted that the attribute information that can be acquired is determined by the GUI accessibility technology to be used and the function of the plug-in.

  FIG. 4 shows a GUI state in an application and an example in which this is expressed using a tree structure. In this example, the GUI state is that the application displays two windows and seven GUI components. FIG. 5 shows a representation format of attribute information held by the nodes in the tree structure in the GUI state representation in FIG. Here, the attribute information of only the node corresponding to the window 1 is shown.

  The attribute information is represented by a set of attribute name / value pairs. In this example, the name, type (window, label, text box, combo box, etc.), process ID, and setting value can be acquired by GUI accessibility technology. When the value is blank, it means that the value for the attribute name is not set (the setting value which is the attribute name in this example has an input value from a user such as a text box as a value. It is blank because there is no input from the user.)

  In the present invention, necessary information is acquired from the application by the GUI accessibility technology, and the expression of the GUI state as in the examples of FIGS. 4 and 5 is recorded in a log so as to be reproducible. A lot of information should be acquired as information to be recorded as a log unless there is a restriction on functions such as GUI accessibility technology. This is because the viewpoint of analysis changes as the log analysis progresses, as described above. However, if information that is obviously not of interest is known in advance, it is possible to specify a GUI component to be excluded from the log acquisition target or its attribute information conditions from the input device 1.

  The condition for specifying the GUI component or its attribute information to be excluded from the log acquisition target is to exclude the GUI component or the window itself from the acquisition target, or to acquire the GUI component or the window itself but exclude only specific attribute information of them. And the contents of the condition that is specific designation information to be excluded. The condition content is information that specifies the relationship between the attribute name and its value when the condition type excludes the GUI part or the window itself from the acquisition target. If the window itself is to be acquired but only specific attribute information is to be excluded, information specifying the relationship between the attribute name and its value, and information specifying the attribute name to be excluded when the relationship is established is there. Here, the relationship can be expressed by general four arithmetic operations, large / small, equal sign, inequality sign, and the like, and the relationship between a plurality of attribute names and their values may be combined with propositional logic symbols. FIG. 6 shows an example of a condition for designating a GUI component or its attribute information to be excluded from the log acquisition target.

  In the initial setting of the direct access function unit 5 in step S21 of the procedure of FIG. 2, conditions for designating such GUI components to be excluded from the log acquisition target or their attribute information are registered. Similarly, the designation of the timing for obtaining the GUI state when using the direct access function input from the input device 1 is also registered. When specifying the acquisition timing of the GUI state when using the direct access function, an acquisition time specification, an interval length for acquiring the GUI state and acquiring the next GUI state, or a user specific operation (for example, a mouse) It is possible to specify acquisition according to click or key down.

  In the initial setting of the event detection function unit 6 in step S22 in the procedure of FIG. 2, a condition for specifying a GUI component to be excluded from the log acquisition target input from the input device 1 or its attribute information is registered in the storage device 2. .

  Further, when an instruction to start log recording is input from the input device 1, the log acquisition control unit 7 acquires the GUI state of the application using the direct access function unit 5 or the event detection function unit 6, and logs And is recorded in the storage device 2. Specifically, the procedure of FIG. 3 is performed.

  First, in step S31, the direct access function unit 5 is used to obtain a tree structure representation of the current GUI state as shown in FIGS. 4 and 5 as an initial state, and a storage device as a log together with the time stamp at the time of acquisition. Output to 2 and record.

  Next, in step S32, in order to record the GUI state change using the event detection function unit 6, here, the value of attribute information other than the frontmost window is changed, or the GUI component other than the frontmost window is changed. A listener that detects an event of generation or disappearance of a GUI component other than the window or the frontmost window is set.

  The steps from S33 to S38 are repeated until the log recording end instruction is input from the input device 1, and the acquired information is output to the storage device 2 as a log and recorded.

  That is, first, in step S34, the specification of the acquisition timing of the GUI state when using the direct access function input in advance is referred to, and it is checked whether or not the current situation matches the specification of the timing. For example, if the timing designation is a time designation, it is checked whether the current time is that time. Also, if the timing is specified as an interval, it is checked whether the time of the specified interval has elapsed since the time when the GUI state was previously acquired. If the timing specification is a user operation specification, it is checked whether an OS event of the user operation has occurred.

  If the timing designation matches the current situation in step S34, the direct access function unit 5 acquires the GUI state of the foremost window in the subsequent step S35. For example, consider the case where the currently displayed GUI is FIG. If the address registration window is the frontmost window, the GUI state acquired in step S35 is only a tree structure with window 1 in FIG. 4 as the root node and attribute information held by each node. As described above, in order not to impair the response performance of the log recording target software, it is necessary to speed up the log acquisition process. If the direct access function unit 5 acquires the state of all the GUI components, the processing time becomes long. Therefore, in the present invention, the state is acquired only for the GUI state of the frontmost window. As a result, the acquisition process can be speeded up. The GUI state of the frontmost window acquired in this way is output as a log together with the time stamp at the time of acquisition. Thereafter, the process proceeds to S36 in order to perform log output by the event detection function unit 6.

  If, in step S34, the designation of the GUI state acquisition timing when using the direct access function is referenced and the current situation does not conform to the timing designation, it is not the timing to output the log. , The log output by the direct access function unit 5 in S35 is skipped, and the process proceeds to S36.

  In step S36, it is checked whether an event that has detected a change in the GUI state has occurred by registering the listener in step S32. If it has occurred, the contents of the event are output as a log in the subsequent step S37. Note that the listener registered in S32 relates to an event of a change in attribute information other than the frontmost window, generation of a GUI part other than the frontmost window, or disappearance of a GUI part other than the window or frontmost window. I want to be. In step S35, the GUI state acquired by the direct access function unit 5 is limited to information on the foreground window in order to speed up the processing. For this reason, it is not possible to acquire information such as a change in the state of the back window or the disappearance of the window. Therefore, in step S37, the event detection function unit 6 supplements these pieces of information. The content of the event to be output is made to be able to reproduce the tree structure representation of the current GUI state by modifying the tree structure representation of the GUI state acquired in S31 according to the content of the output event. There is a need. When a change event is detected, at least information as shown in FIG. 7 is output and recorded according to the type of the event together with the event occurrence time stamp.

  That is, when the type of event is a change of an attribute value other than the frontmost window, information specifying the node in the GUI state expression corresponding to the changed GUI component or window, and the changed attribute name And output the value. For example, it is assumed that a character string “Japan” is input in a text box next to an address in the GUI state of FIG. 4 and it is detected by an event that the GUI state has changed. From that event, log “Part 3” as information to identify the node in the GUI state expression, “Set Value” as the changed attribute name information, and “Japan” as the changed value information. Output to.

  Similarly, when the type of event is generation of a GUI component other than the frontmost window, information for identifying a node corresponding to the generated GUI component and its attribute information are recorded. Further, information on the parent node of the node corresponding to the generated GUI component is output. For example, it is assumed that a confirmation button in the error screen window in the GUI state of FIG. 4 is newly generated, and it is detected by an event that the GUI state has changed. In this case, “part 7” and the attribute information of the confirmation button are output as information for specifying the node in the GUI state expression, and the corresponding button is included in the error screen window. Window 2 "is output.

  Further, when the event type is the disappearance of the GUI component other than the window or the frontmost window, information specifying the node corresponding to the disappeared window or GUI component is output. For example, it is assumed that the error screen window in the GUI state of FIG. 4 disappears and that the GUI state has changed is detected by an event. In this case, “window 2” is output as information for specifying a node corresponding to the disappeared window or GUI component.

  In general, GUI changes with respect to user operations frequently occur in the foreground window. In the present invention, the GUI state of the frontmost window is acquired by the direct access function unit 5 and is not acquired by the event detection function unit 6 (the listener is not registered). As a result, the amount of events to be detected can be greatly reduced. As a result, event discard due to overflow of the event queue can be prevented, and a change in the state of the GUI can be detected accurately.

  If the change event is not detected in step S36, it is not necessary to record the event, and the procedure is continued after skipping step S37.

  As described above, the steps from S33 to S38 are repeated until the log recording end instruction is input from the input device 1.

  In step S31 or S35, when the tree structure representation of the GUI state is acquired by the direct access function unit 5, a condition for specifying a GUI component or its attribute information to be excluded from the log acquisition target is input in advance. If so, the GUI component that matches the condition or its attribute information is not acquired. If the event detected in step S37 is a change related to the GUI component or its attribute information that also meets the above-described conditions, log output is not performed.

  In the above-described embodiment, the example in which the direct access function unit 5, the event detection function unit 6, and the log acquisition control unit 7 are realized on the computer 4 is shown. However, it may be realized by hardware. Each of these devices exists in a distributed manner on a plurality of computers (hardware resources) connected to each other through an appropriate communication path, and can be executed while communicating with each other.

  The software to be recorded by the apparatus of the present invention may operate on the computer 4 or may operate on another computer (hardware resource) connected to each other through an appropriate communication path. .

  Here, as an embodiment of the present invention, a user operation log recording method when the configuration of FIG. 1 is employed will be described.

  First, when using the present invention, a condition for designating a GUI component to be excluded from a log acquisition target or its attribute information is input from the input device 1 in advance. Here, the information of FIG. 6 is input. Further, the timing for acquiring the GUI state by the direct access function unit 5 is specified from the input device 1.

  In actual operation, it is necessary to set the timing in consideration of the performance of the information system terminal (computer) on which the application operates. As described above, acquisition of the GUI state by the direct access function unit 5 may deteriorate the response performance of the application to the user operation if the frequency is increased. Here, it is assumed that the timing is specified so that the direct access function unit 5 acquires the GUI state only when the user clicks the mouse and inputs the keyboard.

  As described above, when various initial parameters are input from the input device 1, the log acquisition control unit 7 in the computer 4 performs the procedure of FIG. 2, registers the input parameters in the storage device 2, and directly accesses the function unit 5. Also, the event detection function unit 6 is initialized.

  Next, when an instruction to start log recording is input from the input device 1, the log acquisition control unit 7 of the computer 4 performs the procedure of FIG. The GUI state of the information system terminal at this time is shown in FIG. 4, and the attribute information that can be acquired by the GUI accessibility technology or the dedicated plug-in is the name, type as shown in the attribute name shown in FIG. The description of the embodiment will be continued on the assumption that the process ID and setting value information.

  First, according to the procedure of FIG. 3, step S31 is performed, and the direct access function unit 5 acquires the GUI state of FIG. 4 from the application as the initial state. In this embodiment, since there is a condition for specifying a GUI component to be excluded from the log acquisition target shown in FIG. 6 or its attribute information, the tree structure representation of the GUI state acquired here is as shown in FIG. Become. However, the process IDs of the windows and GUI parts here are all “1111”. According to the condition 3 in FIG. 6, the nodes of the parts 1 and 5 whose type is the title bar are excluded from acquisition targets. Similarly, it should be noted that the setting value information is excluded from the attribute information of the window 1 and the window 2 whose types are windows according to the condition 2 in FIG. In S31, the information of FIG. 8 and the time stamp at the time of acquisition are recorded in the storage device 2 in the structure data expression format such as XML, for example.

  Subsequently, the log acquisition control unit 7 proceeds to step S <b> 32 in FIG. 3, and sets a listener for event detection using the event detection function unit 6. The listener registers only the event relating to the change of the attribute information value other than the frontmost window, the generation of a GUI part other than the frontmost window, or the disappearance of the GUI part other than the window or the frontmost window. After the listener is set, the steps from S33 to S38 are repeated until a log recording end instruction is input from the input device 1.

  Here, for example, let us consider a case where in the current GUI state of FIG. 4, the user presses the confirmation button of the error screen window, and the error screen window disappears and the GUI state as shown in FIG. At this time, since the log acquisition timing by the direct access function unit 5 is designated as mouse click, the log acquisition control unit 7 executes steps S34 and S35 of FIG.

  In step S35, the direct access function unit 5 is used to obtain the tree structure representation of the GUI state of the frontmost window in FIG. 9, and the tree structure representation of the GUI state in XML format or the like together with the time stamp at the time of acquisition. Record in the storage device 2. FIG. 10 shows a tree structure representation of the GUI state acquired here. Since only the GUI state of the foreground window is acquired here, the log acquisition process can be performed at high speed. On the other hand, it should be noted that from the information acquired here, it cannot be determined whether the error screen window that has been the forefront window has disappeared or has been placed on the back side. In order to be able to determine these, the event detection function unit 6 is used in the subsequent step of S36 in FIG.

  At this time, if the event detection function unit 6 can correctly detect that the error confirmation window has disappeared, the detected event information is recorded in the storage device 2 together with the time stamp at the time of event detection according to the contents of FIG. To do. The contents of the event recorded in the storage device 2 are shown in FIG. 11 (the time stamp is not particularly limited as long as it is information that can specify the order of event occurrence and GUI status acquisition). Actually, the event of disappearance of the title bar of the error confirmation window also occurs at the same time. However, since it matches the condition 3 in FIG. 6 which is a condition for specifying the GUI component to be excluded from the log acquisition target or its attribute information, the recording is performed. Exclude from the target.

  Note that if the event can be detected accurately, the current GUI state can be reproduced by reflecting the event information on the initial GUI state acquired in S31.

  For example, with respect to the tree structure representation of the initial GUI state of FIG. 8, by deleting the nodes of window 2, component 6, and component 7 and their attribute information from the event contents acquired in FIG. It is possible to obtain a tree structure representation of exactly the same GUI state.

  Since the event can detect only information related to a change in the GUI state, the event has a feature that the GUI state can be reproduced in detail with a small amount of data. However, the problem is that there are cases in which events cannot be detected due to the occurrence of a large number of events even though the GUI state has changed as described above.

  In the present invention, the GUI state of the foreground window where the display changes frequently and events frequently occur is acquired by the direct access function unit 5, and other GUI changes are acquired by the event detection function unit 6. Therefore, it is possible to limit the events to be detected (limit listeners to be registered), and to prevent omission of event detection due to a large number of event detections.

  Next, consider a case where the user inputs the keyboard “Japan” into the text box next to the address and enters the GUI state as shown in FIG. In this case, the log acquisition control unit 7 performs steps S34, S35, S36, and S38 in the procedure of FIG. In step S35 in which the designation of the timing for obtaining the state using the direct access function unit 5 is a keyboard input, the state of the GUI of the foreground window is obtained. The acquired information is shown in FIG. With the change as shown in FIG. 12, the attribute value change event of the GUI component of the frontmost window is originally generated, but the listener registered in step S32 is limited to the change of the attribute value of the GUI component of the window other than the frontmost window. Has been. Therefore, no event occurs in step S36, and no event detection by the event detection function unit 6 is performed in step S37. As a result, the information in FIG. 13 is recorded in the storage device 2 for the keyboard input here.

  Finally, for the GUI state of FIG. 12, the user clicks the registration button with the mouse, and a new GUI component labeled “Registration complete” is generated in the address registration window as shown in FIG. The operation of the log acquisition control unit 7 when the result window is displayed on the front will be described.

  At this time, steps S34, S35, S36, and S37 of the procedure of FIG. 3 are performed. Since the mouse is clicked, the GUI state of the frontmost window is acquired by the direct access function unit 5 in S35. Here, the tree structure representation of FIG. 15 for the result window is stored in the storage device 2 together with the time stamp at the time of acquisition. To be recorded. Further, the event of the GUI component of the label newly generated in the rear window is detected in the subsequent event of S36, and the information of FIG. 16 is recorded in the storage device 2 by the step of S37.

  As in the present embodiment, the present invention can speed up the acquisition process by limiting the GUI state acquisition by the direct access function to the frontmost window, and can detect only the state change of the GUI of the rear window. It is possible to prevent omission of detection due to a large amount of event detection. As described above, it is possible to record an accurate GUI state change without deteriorating the response performance of the log acquisition target software.

  1: input device, 2: storage device, 3: display device, 4: computer, 5: direct access function unit, 6: event detection function unit, 7: log acquisition control unit.

K. D. Fenstermacher and M. Ginsburg, "A Lightweight Framework for Cross-Application User Monitoring", IEEE Computer, Vol.35, Issue 3, pp. 51-59, 2002 Takahashi, Mouri, Oda, Iwayama, "App Operation Log Estimation Method for PC Work Visualization", IPSJ 50th Anniversary (72nd) National Conference Proceedings, pp. 4-541-4-542, 2010 “Outline of UI Automation”, [online], Microsoft Corporation, [searched July 5, 2011], Internet <URL: http://msdn.microsoft.com/en-us/library/ms747327(v=vs .80) .aspx> “Java IDL”, [online], Sun Microsystems, Inc., [October 14, 2011 search], Internet <URL: http://java.sun.com/j2se/1.4/ja/docs/ja/ guide / access / index.html> “Hitachi Universal Design”, [online], Hitachi, Ltd., [October 14, 2011 search], Internet <URL: http://www.hitachi.co.jp/universaldesign/ria/ajax/wai -aria / index.html>

Claims (5)

Log of user operation for GUI provided by the log recording target software operating on an arbitrary computer via the input device and the display device of the arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path The log display target software displays a GUI display state as a GUI on the screen of the display device and a GUI state represented by the display state of the GUI component in the window. A method of recording in a storage device of another computer connected to a computer through an arbitrary communication path,
The arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path,
Registers in the storage device the conditions for specifying the GUI component or its attribute information to be excluded from the log acquisition target and the timing for acquiring the GUI status of the log recording target software using the direct access function. A first step to:
A second step of acquiring a current GUI state as an initial state using a direct access function in response to a log recording start instruction input from an input device, and outputting and recording the log to a storage device;
And a third step of setting a listener for detecting a GUI state change event other than the frontmost window or a window extinction event by an event detection function,
After the third step, the process is repeated until the log recording end instruction is input from the input device.
A fourth step of outputting and recording the GUI state of the foremost window as a log to the storage device in accordance with the designation of the timing for acquiring the GUI state of the log recording target software using the direct access function;
A fifth step of checking whether a change event of the GUI state other than the frontmost window or a window extinction event is detected by the event detection function, and outputting the event information to the storage device as a log and recording it. A log recording method of user operations as a feature. In the fourth step,
Along with the time stamp information, the direct access function is used to obtain the GUI state of the frontmost window directly from the application, the window or GUI component is a node, the inclusion relationship is expressed as a parent-child relationship between the nodes, and each node Output in an expression format having attribute information, and exclude from the output nodes or attribute information that match the conditions for specifying the GUI component to be excluded from the input target of the log or the attribute information. Item 2. A user operation log recording method according to Item 1. In the fourth step,
The frequency of log output / recording can be changed by specifying the time, the interval, or the user operation when specifying the timing for acquiring the application GUI status using the direct access function. Item 2. A user operation log recording method according to Item 1. In the fifth step,
Time stamp information at the time of event detection, change of attribute information value other than the frontmost window of the event detected by the event detection function, generation of GUI parts other than the frontmost window, or GUI other than the window or frontmost window Along with information on the event type that indicates which of the parts disappeared,
If it is a change event of the value of attribute information other than the frontmost window, the information specifying the node in the tree structure representation of the GUI state corresponding to the change target, the attribute name to be changed and the value after the change,
Further, if it is a generation event of a GUI component other than the frontmost window, information for specifying a node in the tree structure representation of the GUI state corresponding to the generation target, attribute information for the generation target, and information for specifying the parent node,
Further, if the event is a change event of disappearance of a GUI component other than the window or the frontmost window, information for specifying a node in the tree structure representation of the GUI state corresponding to the disappearance target is output. The user operation log recording method described in 1. Log of user operation for GUI provided by the log recording target software operating on an arbitrary computer via the input device and the display device of the arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path The log display target software displays a GUI display state as a GUI on the screen of the display device and a GUI state represented by the display state of the GUI component in the window. A device for recording in a storage device of another computer connected to the computer through an arbitrary communication path,
The next device provided on the arbitrary computer or another computer connected to the arbitrary computer through an arbitrary communication path, that is, an input device;
A storage device;
A direct access function unit for directly accessing the log recording target software and obtaining a GUI state or a GUI state of only the frontmost window;
An event detection function unit for setting a listener that detects a GUI state change event other than the frontmost window or a window disappearance event, and detects these events that occur in response to a GUI state change;
When the designation of the GUI component to be excluded from the log acquisition target or the attribute information thereof and the specification of the timing for acquiring the GUI status of the log recording target software using the direct access function unit are input from the input device Is registered in the storage device, and when an instruction to start log recording is input from the input device, the current GUI state is acquired as an initial state using the direct access function unit, and is output to the storage device as a log for recording. Then, a listener for detecting a GUI state change event other than the frontmost window or a window disappearance event is set using the event detection function unit, and thereafter, an instruction to end log recording is input from the input device. According to the specification of the timing for acquiring the GUI status of the log recording target software using the direct access function unit The GUI status of the frontmost window is output to the storage device as a log and recorded. Further, the event detection function unit checks whether a change event of the GUI status other than the frontmost window or a window disappearance event is detected. A log recording device for user operation, comprising: a log acquisition control unit that outputs and records information as a log to a storage device.

Images