JP2013239155A - Operation recording apparatus, operation record method and operation record program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation recording apparatus which does not need implementation of event detection processing for each GUI element of a GUI application.SOLUTION: An operation recording apparatus that records a user operation in a GUI application including one or more GUI elements, includes event detection means for detecting a predetermined type of event, of the events detected in the GUI element, event identifying means that uses information on a GUI element type for identifying an event relating to the user operation from the detected events, and event information recording means for recording information on the identified event.

Description

  The present invention relates to an operation recording device, an operation recording method, and an operation recording program, and more particularly to an operation recording device, an operation recording method, and an operation recording program for recording user operations in a GUI application.

  In an application using a GUI (Graphical User Interface), there is a mechanism for acquiring and storing a history of user operations. User operations are obtained by detecting events that occur in the GUI elements that make up the application (for example, clicking the left mouse button on the “Confirm” button, keyboard input operations in a text box, etc.) can do. The user operation history acquired in this way is used, for example, to analyze an application problem caused by the user operation.

  Patent Document 1 discloses an event control apparatus that detects an event that has occurred in a computer system and stores an event whose event ID or event message satisfies a predetermined condition.

  Patent Document 2 discloses a mechanism for storing user operations such as application activation, window opening, or character input together with the application or window ID.

Japanese Patent No. 3418034 JP 2004-21438 A

  However, in the existing method, in order to detect an event that occurs in the GUI element, it is necessary to implement a process for detecting and recording the event in each GUI element. For this reason, every time a GUI element is added or changed, it is necessary to add or change such processing, which is costly to implement.

  The present invention has been made in view of such a problem, and an object thereof is to provide an operation recording apparatus that does not need to implement an event detection process for each GUI element of a GUI application.

In order to solve the above-described problems and achieve the object, an operation recording apparatus according to an embodiment of the present invention is provided.
An operation recording apparatus for recording a user operation in a GUI application including one or more GUI elements,
Event detection means for detecting a predetermined type of events detected in the GUI element;
Event specifying means for specifying an event related to the user's operation from the detected event using information on a type of GUI element;
Event information recording means for recording information of the identified event;
Have

In addition, an operation recording method according to an embodiment of the present invention includes:
An operation recording method for recording a user operation in a GUI application including one or more GUI elements,
An event detection step of detecting a predetermined type of events detected in the GUI element;
An event identification step of identifying an event related to the user's operation from the detected event using information on a type of GUI element;
An event information recording step for recording information of the identified event;
Have

The operation recording program in one embodiment of the present invention is:
An operation recording program for causing a computer to execute an operation recording method for recording a user operation in a GUI application including one or more GUI elements,
An event detection step of detecting a predetermined type of events detected in the GUI element;
An event identification step of identifying an event related to the user's operation from the detected event using information on a type of GUI element;
An event information recording step for recording information of the identified event;
Is an operation recording program that causes a computer to execute.

  ADVANTAGE OF THE INVENTION According to this invention, the operation recording apparatus which does not need to mount an event detection process for every GUI element of a GUI application can be provided.

The figure showing the example of the hardware constitutions of the operation recording apparatus in one Embodiment of this invention. The figure showing the example of the functional block of the operation recording apparatus in one Embodiment of this invention. The figure showing the example of the element tree of the GUI element which comprises a GUI application. The flowchart showing the process at the time of starting of the GUI application by the operation recording apparatus in one Embodiment of this invention. The flowchart showing the event detection process by the operation recording device in one Embodiment of this invention. The flowchart showing the recording process of the operation history by the operation recording apparatus in one Embodiment of this invention. The flowchart showing the recording process of the operation history by the operation recording apparatus in one Embodiment of this invention. The figure showing the example of the event extraction condition definition table which the operation recording device in one Embodiment of this invention uses. The figure showing the example of the object element type definition table which the operation recording device in one embodiment of the present invention uses. The figure showing the example of the object element table which the operation recording device in one Embodiment of this invention uses. The figure showing the example of the operation history information which the operation recording device in one embodiment of the present invention outputs. The figure showing the example of a display of a GUI application. The figure showing the example of the XAML file of a GUI application. The flowchart showing the process at the time of starting of the GUI application by the operation recording apparatus in one Embodiment of this invention. The flowchart showing the event detection process by the operation recording device in one Embodiment of this invention. The figure showing the example of the object element name definition table which the operation recording device in one Embodiment of this invention uses. The figure showing the example of the object element type definition table which the operation recording device in one embodiment of the present invention uses. The figure showing the example of the object element table which the operation recording device in one Embodiment of this invention uses. The figure showing the example of the operation history information which the operation recording device in one embodiment of the present invention outputs.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1. Overview 2. 2. Hardware configuration Functional configuration 4. Process Flow 4.1 Process Flow at GUI Application Startup 4.2 Event Detection Process Flow 4.3 Operation History Recording Process Flow Application examples

(1. Overview)
First, the outline of the configuration of the operation recording apparatus according to an embodiment of the present invention will be described. The operation recording device and the GUI application that is the operation recording target can be configured using a WPF (Windows Presentation Foundation) framework developed by Microsoft (registered trademark).

  In the WPF, GUI elements constituting a GUI application form a tree structure 300 (hereinafter referred to as “element tree”) as shown in FIG. 3, for example. In the example of FIG. 3, the root element 310 (root node) is a “Window” element, and “Canvas” and “Grid” elements exist as its child elements (child nodes). Under the “Grid” element, there are further “Button” elements and “Checkbox” elements as child elements. In WPF, two types of events, a bubble event and a tunnel event, occur in response to user input performed on these GUI elements. When a bubble event occurs, the WPF first calls an event handler registered in the GUI element that generated the event, and sequentially calls the event handlers of the parent element until the root element 310 of the element tree 300 is reached. On the other hand, when a tunnel event occurs, the WPF first calls an event handler registered in the root element 310. After that, according to the element tree 300, the event handlers of the child elements are sequentially called up to the event generation source element. The event handler called in response to the occurrence of these events receives event-accompanying information including information on the GUI element that is the event generation source and properties (attribute information) indicating the state of the GUI element from the WPF. The element tree 300 can be acquired from a file in the XAML (Extensible Application Markup Language) format included in the GUI application by WPF.

  The operation recording apparatus according to the embodiment of the present invention registers an event handler that performs an event detection process on the root element 310 of the element tree 300, thereby adding an event incident to a tunnel event in all the child elements constituting the element tree. Information can be acquired. As a result, it is not necessary to implement an event detection process for each GUI element constituting the element tree.

(2. Hardware configuration)
The hardware configuration of the operation recording apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. The operation recording device 100 illustrated in FIG. 1 includes a CPU 102, a ROM 104, a RAM 106, an HDD 108, a communication device 110, a display device 112, an input device 114, and a bus 116.

  The CPU 102 executes a system program for controlling the operation of the operation recording apparatus 100 and a user application. The ROM 104 stores a system program executed by the CPU 102 and the like. The RAM 106 constitutes a work area for the CPU 102. The HDD 108 stores system programs, user applications, user data, and the like. The communication device 110 includes, for example, a wired or wireless communication interface and its control device, and is used to communicate with an external device. The display device 112 includes, for example, a display and a graphics controller, and presents information to the user. The input device 114 receives input from a user such as a keyboard, a mouse, and a touch panel. The bus 116 connects the above-described devices constituting the operation recording device 100 to each other and exchanges data.

  With the above configuration, the operation recording apparatus 100 according to an embodiment of the present invention can provide an operation recording apparatus that does not need to implement an event detection process for each GUI element of a GUI application.

(3. Functional configuration)
Next, functional blocks of the operation recording apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. FIG. 2 shows elements particularly relevant to the description of the present embodiment among various elements included in the operation recording apparatus 100. 2 includes an input unit 200, an event notification unit 202, a registration unit 204, a target event management unit 206, an event detection unit 208, an event identification unit 210, and a target element management unit 212. Operation history recording means 214 and display means 216.

  The input unit 200 receives an input made by the user using the GUI application through the input device 114. Information on the input (for example, which button of the mouse has been clicked, which key input has been made, etc.) is notified to the event notification means 202 through the OS (Operating System).

  The event notification unit 202 notifies the event detection unit 208 of the input from the user in the input unit 200 as an input event. The notification is performed when the event notification unit 202 calls the event detection unit 208 as an event handler registered in the root element 310 by the registration unit 204 described later. In addition, the event notification unit 202 passes event-accompanying information to the event detection unit 208 together with the notification. The event notification unit 202 can be realized by using a WPF function.

The registration unit 204 registers the event detection unit 208 in the event notification unit 202 as an event handler. More specifically, the registration unit 204 registers the event detection unit 208 with respect to the root “Window” of the element tree so that an event regarding the event type specified by the target event management unit 206 described later can be acquired. . The event handler registration procedure in WPF is as follows.
Element object. Event type + = Event handler An element object represents an object (instance) of a GUI element. In the present embodiment, as shown in FIG. 3, an object of “Window” that is the root element 310 is designated. The event type is a type of event that is defined in advance according to the mode of input from the user, and includes, for example, “PreviewMouseLeftButtonDown”, “PreviewKeyDown”, and the like. “PreviewMouseLeftButtonDown” represents the type of tunnel event that occurs when the left mouse button is clicked. “PreviewKeyDown” represents a tunnel event that occurs when a keyboard button is pressed. In the present embodiment, the type of tunnel event is specified in the root element 310 so that all the events of all the child elements constituting the element tree 300 can be detected. In the present embodiment, the event handler corresponds to the event detection unit 208. The registration unit 204 registers the event detection unit 208 for the event type specified by the target event management unit 206 described later.

  The target event management unit 206 specifies an event type to be specified when the registration unit 204 registers an event handler. The event type is defined by, for example, an event extraction condition definition table 800 shown in FIG. The event extraction condition definition table 800 shown in FIG. 8 includes items of event ID, event type, and event notation name. The event ID is an ID assigned to specify the type of the detected event. The event type represents the type of event to be detected. The event name is a name for making the event type easy to understand for the user. The event notation name will be used later when an operation history is recorded. The registration unit 204 described above can detect event information of the event type specified by the target event management unit 206. Note that the event extraction condition definition table 800 may be defined in advance by the user. Further, the target event management unit 206 may add or delete the contents described in the event extraction condition definition table 800 as necessary.

  The event detection unit 208 receives the notification from the event notification unit 202 and detects an event of the event type specified by the target event management unit 206. The event detection unit 208 also receives event-accompanying information including information on the event generation source element and a property indicating the state of the GUI element from the event detection unit 204. For example, in WPF, the event detection unit 208 that is an event handler receives a reference from the event detection unit 204 to an object that includes event-accompanying information as an argument. The received event incidental information is passed to the event identification unit 210.

  The event identification unit 210 receives event-accompanying information from the event detection unit 208. If the type of the GUI element that is the event generation source of the received event-accompanying information is the type of the GUI element specified by the target element management unit 212 described later, predetermined property information is acquired from the event-accompanying information. For example, when the element type is “System.Windows.Controls.Button”, the predetermined property information may be an object name of a GUI element and a button label. The property information to be acquired is arbitrary. Property information to be acquired is specified by the target element management unit 212 described later. The acquired property information is passed to the operation history recording unit 214 as operation history information together with event occurrence date / time information, an element notation name of the element type, and an event notation name of the identified event. Here, in the event occurrence date and time information, for example, the date and time when the event detection unit 208 detects the event can be stored. Event occurrence date / time information can be acquired at an arbitrary timing. The element notation name is an element notation name of the identified event generation element, and can be acquired by referring to a target element table 1000 (FIG. 10) described later, for example. The event notation name is the event notation name of the identified event, and can be acquired by referring to the event extraction condition definition table 800 in FIG. 8, for example. Note that if the event-accompanying information received from the event detection unit 208 does not include information about the type of the element specified by the target element management unit 212, no information is passed to the operation history recording unit 214.

  The target element management unit 212 identifies the type of the GUI element that is the target of the operation history recording and the property that represents the state of the GUI element to be acquired. The type of GUI element is defined by a target element table 1000 shown in FIG. 10, for example. The target element table 1000 shown in FIG. 10 includes items of element ID, element type, element notation name, acquisition property 1, and acquisition property 2. The element ID is an ID given to specify the element type. The element type is a type of GUI element that is a target for which information is to be acquired. The element notation name is a notation name that clearly represents the GUI element. The element notation name is used later when a history of user operations is recorded. Acquisition properties 1 and 2 indicate properties to be acquired from the GUI element. Properties that can be acquired from the GUI element differ depending on the type of the GUI element. In the present embodiment, the types and number of properties that can be specified are arbitrary. The event specifying unit 210 described above specifies the type of the event source element from the received event incidental information, and refers to the element type item of the target element table 1000, so that the event incidental information is stored as an operation history. It is determined whether the information is to be recorded. The target element table 1000 shown in FIG. 10 is automatically generated based on the target element type definition table 900 shown in FIG. The target element type definition table 900 is a table in which element types that are defined in advance by the user and for which information is to be acquired are defined. The target element management unit 212 extracts only the element types included in the GUI application that is the target of operation recording from the element types defined in the target element type definition table 900 in advance, and generates the target element table 1000. Note that the type of the GUI element included in the GUI application can be acquired as information of the element tree 300 expressed in the XAML format.

  The operation history recording unit 214 records the operation history information received from the event specifying unit 212. FIG. 11 shows an example in which recorded operation history information is stored in a table format. Among the stored items, the event occurrence date / time, element notation name, and event notation name are as described above. The acquired properties 1 and 2 store the property values specified by the target element table 1000 (FIG. 10) and acquired by the event specifying unit 210. For example, in the record related to the element notation name “button” in FIG. 11, the value of the “Name” property acquired from the element type “System.Windows.Controls.Button” in FIG. 10 is “button1”, and the “Label” property The value of indicates “confirmation”. As will be described in detail later, the operation history recording unit 214 can take two forms of the operation history recording process: a performance-oriented method and a certainty-oriented method. In the former method, the operation history recording process command is accumulated in a queue arranged on the memory, and the recording process accumulated in the queue is executed from another thread as needed. On the other hand, in the latter method, operation history records are sequentially output. The operation history recording unit 214 records the operation history information by an arbitrary method such as writing to a file format or a database.

  The display unit 216 displays the operation history stored by the operation history recording unit 214 on the display device 112 according to an instruction from the user.

  With the above configuration, the operation recording apparatus 100 according to an embodiment of the present invention does not need to implement a process of recording operation history information for each GUI element of the GUI application, which contributes to a reduction in development cost. In addition, by making it possible to define event extraction conditions in advance by the user, it is possible to record information on operation history desired by the user. In particular, the user can add or change the definition of the event extraction condition at any time, and can selectively record only a desired event.

  The event specifying unit 210 and the operation history recording unit 214 may be realized as part of the event detection unit 208.

(4. Processing flow)
Next, a processing flow of the operation recording apparatus 100 according to an embodiment of the present invention will be described with reference to FIGS.

(4.1 Processing flow at GUI application startup)
First, the processing flow when starting up the GUI application will be described with reference to FIG.

  In step S400, when a GUI application that is an operation history recording target is activated, the processing starts.

  In step S <b> 402, the target event management unit 206 reads the event extraction condition definition table 800 (FIG. 8) defined in advance by the user, and identifies an event (tunnel event) that is a target of operation history acquisition.

  In step S <b> 404, the registration unit 204 registers an event handler (event detection unit 208) in the root element 310 of the element tree 300 for the event type of the event specified by the target event management unit 206.

  In step S406, the target element management unit 212 collates the target element type definition table 900 (FIG. 9) defined in advance by the user with the element tree 300 of the GUI elements constituting the GUI application. Then, a GUI element to be a target of the operation history is extracted from the GUI elements actually included in the GUI application, and the target element table 1000 (FIG. 10) is generated.

  As described above, the operation recording apparatus 100 according to the embodiment of the present invention can detect an event type event designated in advance by the user among events generated in the GUI application. In addition, among the GUI elements designated in advance by the user, it is possible to detect an event that has occurred in the GUI elements constituting the GUI application.

(4.2 Event detection process flow)
Next, the event detection process flow will be described with reference to FIG.

  In step S500, when the user performs an input operation through the input unit 200 on the GUI application, the process starts.

  In step S502, the event notification unit 202 notifies an event (tunnel event) corresponding to an input from the user to the event detection unit 208 (event handler registered for each event type of the root element 310). At this time, the event notification unit 202 passes event-accompanying information including information on the event generation source element and a property indicating the state of the GUI element to the event detection unit 208.

  In step S504, the event detection unit 208 detects an event and receives event-accompanying information. The received event incidental information is passed to the event identification unit 210.

  In step S506, the event specifying unit 210 acquires information about the type of the GUI element that is the event generation source from the event-accompanying information.

  In step S508, the event specifying unit 210 compares the type of the event generation source element with the element type of the target element table 1000 (FIG. 10), and checks whether there is a corresponding one. As a result, if there is a corresponding item, the process proceeds to step S510. If there is no corresponding item, the process advances to step S514 to end the process.

  In step S510, the event specifying unit 210 acquires the property information of the event generation source element from the event incidental information. Property information to be acquired is defined by the target element table 1000 (FIG. 10), and differs depending on the type of the event generation source element. Further, the event specifying unit 210 acquires event occurrence date / time information, an element notation name of an event occurrence source element, and an event notation name of the specified event. Then, the event occurrence date / time information, the element notation name of the event occurrence source element, the event notation name of the identified event, and property information are passed to the operation history recording unit 214 as operation history information.

  In step S512, the operation history recording unit 214 records the information received from the event specifying unit 210 as an operation history. An example of a table for storing the recorded operation history is shown in FIG. Here, the operation history recording means 214 can take two forms of the operation history recording process, a performance-oriented method and a certainty-oriented method. Each type of processing will be described later.

  As described above, the operation recording apparatus 100 according to the embodiment of the present invention can acquire the operation history based on the event type and the element type specified by the user. The user can grasp the actual usage status of the application using the recorded operation history. Further, the user can investigate the cause when a problem occurs in the application and reproduce the problem by using the recorded operation history.

(4.3 Operation history recording process flow)
Next, an operation history recording process flow will be described with reference to FIGS. As described above, the operation history recording unit 214 can selectively use two types of implementation methods, namely, a performance-oriented method and a certainty-oriented method with respect to operation history recording processing. The recorded operation history may be presented to the user later by the display means 216.

  FIG. 6 shows a processing flow in the performance-oriented method. In this method, the operation history recording unit 214 records the operation history by parallel processing using two threads.

  In step S600, processing by the first thread of the operation history recording unit 214 is started.

  In step S602, the operation history recording unit 214 records the operation history information recording processing command received from the event specifying unit 210 in the operation history recording queue arranged in the memory. The recording process is processed in parallel in a second thread (steps S650 to S658) described later.

  In step S604, the process ends.

  In step S650, processing by the second thread of the operation history recording unit 214 starts.

  In step S652, the operation history recording unit 214 reads the operation history recording queue.

  In step S654, if the operation history recording queue is empty, the process proceeds to step S658, and the process ends. If the operation history record queue is not empty, the process proceeds to step S656.

  In step S656, the operation history recording unit 214 executes the operation history information recording process recorded in the operation history recording queue and outputs the operation history. FIG. 11 shows an example in which recorded operation history information is stored in a table format. Thereafter, the process returns to step S652, and the operation history record queue is read again.

  FIG. 7 shows a processing flow in the method of emphasizing certainty. In this method, the operation history recording unit 214 sequentially outputs operation history information in a single thread.

  In step S700, the process starts.

  In step S702, the operation history recording unit 214 outputs the operation history information received from the event specifying unit 210 as an operation history. FIG. 11 shows an example in which recorded operation history information is stored in a table format.

  In step S704, the process ends.

  As described above, since the operation recording apparatus 100 according to the embodiment of the present invention records an event designated in advance by the user, it is possible to reduce a data capacity of an operation history necessary for recording. Further, since an event designated in advance by the user is recorded, it is not necessary to perform unnecessary computer processing, and operation history information can be recorded without imposing a load on the application. Also, two types of operation history recording methods can be selectively used according to the performance of the operation recording device 100 and the characteristics of the application that records the operation.

  Although the present embodiment has been described by taking a GUI application corresponding to WPF as an example, the present invention is not limited to that example.

(5. Application examples)
As described above, the operation recording apparatus 100 according to the embodiment of the present invention performs an event detection process for each GUI element of a GUI application by acquiring an operation history based on information on an event type and a GUI element type. No need to implement.

  Here, FIG. 12 shows a display example of the screen 150 by a general GUI application. 12 includes a registration button 151, a cancel button 152, text boxes 153 and 154, and labels 155 and 156. An example of the contents of the XAML file corresponding to this screen 150 is shown in FIG. Consider obtaining an operation history under the following conditions on the screen 150 using the operation recording device 100 described above.

Element type: "System.Windows.Controls.Buton" (see Figure 9)
Event: “PreviewMouseLeftButtonDown” (See Figure 8)
As a result, the operation recording apparatus 100 acquires an operation history related to the registration button 151 and the cancel button 152.

  In such a case, the user may desire to obtain an operation history related to only one of the buttons. Therefore, hereinafter, an operation recording apparatus 100A according to an application example for responding to such a request will be described. The hardware configuration of the operation recording apparatus 100A is the same as the example shown in FIG. The functional block diagram of the operation recording apparatus 100A is the same as the example shown in FIG. 2, but the processing contents of some means are different. Therefore, the following description will be given focusing on the difference.

(Functional configuration)
The target element management unit 212 of the operation recording apparatus 100 that has already been described uses the target element table 1000 as described with reference to FIG. The target element table is generated based on the target element type definition table 900 of FIG.

  On the other hand, in the target element management unit 212 of the operation recording apparatus 100A in the application example, the target element table is generated based on the target element type definition table 900 and the target element name definition table 910 illustrated in FIG. .

  The target element name definition table 910 is used to further narrow down the elements from which operation history is acquired using the element type and the element name. The target element name definition table 910 in FIG. 16 includes items of element name ID, element name, and element type. The element name ID is an ID assigned to each item. The element name is a name given to the GUI element, and is indicated by a “Name” property in the XAML file of FIG. The element type represents the type of the GUI element. Similar to the target element type definition table 900, the target element name definition table 910 is generated in advance by the user.

  The target element name definition table 910 illustrated in FIG. 16 is created corresponding to the screen 150 in FIG. 12 and the XAML file in FIG. The target element name definition table 910 is generated for the purpose of obtaining an operation history for an element whose element name is “SubmitBtn” among elements of the element type “System.Windows.Controls.Button”. That is, the operation history for the element name “CancelBtn” of the same element type is not an acquisition target.

  18 is a target element table 1010 generated based on the target element name definition table 910 in FIG. 16 and the target element type definition table 900 in FIG. Here, the target element type definition table 900 of FIG. 17 has the same items as the target element type definition table 900 of FIG. 9, but partially includes different data. The target element table 1010 in FIG. 18 has a new “element name” item as compared with the target element table 1000 in FIG. This item corresponds to the item “element name” in the target element name definition table 910 of FIG.

  The target element management unit 212 of the operation recording apparatus 100A collates the target element name definition table 910 with the element tree 300, and uses the element names defined in the target element name definition table 910 from the elements included in the element tree 300. Extract elements that have. Then, the extracted element is further checked against the target element type definition table 900, and the element type matching is extracted to generate the target element table 1010.

  The target element table 1010 of FIG. 18 includes an item of an element whose element name is “SubmitBtn” among elements of the element type “System.Windows.Controls.Button”, but the element name included in the XAML file of FIG. Does not include items with the element “CancelBtn”.

  The generated target element table 1010 is later referred to when an event is detected on the screen 150. When the element name and element type of the event are defined in the target element table 1010, the GUI element Is recorded. Accordingly, in the example of FIG. 18, only the operation history related to “SubmitBtn” is acquired from the elements of the element type “System.Windows.Controls.Button”, and “CancelBtn” is not acquired.

(Processing flow when starting GUI application)
FIG. 14 corresponds to FIG. 4 and illustrates a processing flow when the operation recording apparatus 100A starts up the GUI application.

  In step S700, when a GUI application that is an operation history recording target is activated, the processing starts.

  In step S <b> 702, the target event management unit 206 reads the event extraction condition definition table 800 (FIG. 8) defined in advance by the user, and identifies an event (tunnel event) that is a target of operation history acquisition.

  In step S <b> 704, the registration unit 204 registers an event handler (event detection unit 208) in the root element 310 of the element tree 300 for the event type of the event specified by the target event management unit 206.

  In step S706, the target element management unit 212 compares the target element name definition table 910 (FIG. 16) with the element tree 300, and from the elements included in the element tree 300 to the target element name definition table 910 (FIG. 16). Extract an element with a defined element name. In the example of FIG. 16, for example, an element of the element type “System.Windows.Controls.Button” is extracted. Then, the extracted elements are further collated with the target element type definition table 900 (FIG. 17), and the elements having the same element type are extracted to generate the target element table 1010 (FIG. 18). Since FIG. 17 includes an item of the element type “System.Windows.Controls.Button”, the target element table 1010 including the same item of FIG. 17 is generated.

  As described above, the operation recording apparatus 100A according to the embodiment of the present invention can detect an event type event designated in advance by the user among events generated in the GUI application. Further, only the operation history related to the GUI element having a name designated in advance by the user can be recorded.

  If the target element name definition table 910 has not been generated by the user, the same processing as step S406 in FIG. 4 may be performed in step S706.

(Event detection process flow)
FIG. 15 is a flowchart of an event detection process flow of the operation recording apparatus 100A according to the embodiment of the present invention. The flowchart of FIG. 15 corresponds to the flowchart of FIG. 5 already described.

  In step S800, when the user performs an input operation through the input unit 200 on the GUI application, the process starts.

  In step S802, the event notification unit 202 notifies an event (tunnel event) corresponding to the input from the user to the event detection unit 208 (event handler registered for each event type of the root element 310). At this time, the event notification unit 202 passes event-accompanying information including information on the event generation source element and a property indicating the state of the GUI element to the event detection unit 208.

  In step S804, the event detection unit 208 detects an event and receives event-accompanying information. The received event incidental information is passed to the event identification unit 210.

  In step S806, the event identification unit 210 acquires information about the type of the GUI element that is the event generation source from the event-accompanying information.

  In step S808, the event specifying unit 210 compares the type of the event generation source element with the element type of the target element table 1010 (FIG. 18), and checks whether there is a corresponding one. As a result, if there is a corresponding item, the process proceeds to step S810. If there is no corresponding item, the process proceeds to step S814 to end the process.

  In step S810, the event identification unit 210 acquires the property information of the event generation source element from the event incidental information. The property information to be acquired is defined by the target element table 1010 and differs depending on the type of the event generation source element.

  Further, the event specifying unit 210 acquires event occurrence date / time information, an element notation name of an event occurrence source element, and an event notation name of the specified event. Then, the event occurrence date / time information, the element notation name of the event occurrence source element, the event notation name of the identified event, and property information are passed to the operation history recording unit 214 as operation history information.

  In step S812, the operation history recording unit 214 records the information received from the event specifying unit 210 as an operation history. An example of a table for storing the recorded operation history is shown in FIG.

  The operation history information 1100 in FIG. 19 stores only the operation history for the registration button 151 among the two buttons on the screen 150 in FIG. Acquisition property 1 (“SubmitBtn”) and acquisition property 2 (“registration”) are the values of the properties (“Name” and “Content”) specified in the target element table 1010 of FIG. Here, as the acquisition property 1, the element name used in FIG. 16 is output as a history.

  As described above, the operation recording apparatus 100A according to the embodiment of the present invention records an event designated in advance by the user, and thus it is possible to reduce the data capacity of the operation history necessary for recording. Further, since an event designated in advance by the user is recorded, it is not necessary to perform unnecessary computer processing, and operation history information can be recorded without imposing a load on the application.

100 operation recording device 200 input unit 202 event notification unit 204 registration unit 206 target event management unit 208 event detection unit 210 event identification unit 212 target element management unit 214 operation history recording unit 216 display unit

Claims (9)

An operation recording apparatus for recording a user operation in a GUI application including one or more GUI elements,
Event detection means for detecting a predetermined type of events detected in the GUI element;
Event specifying means for specifying an event related to the user's operation from the detected event using information on a type of GUI element;
Event information recording means for recording information of the identified event;
An operation recording apparatus. The event specifying means specifies the event by further using information on the name of the GUI element.
The operation recording apparatus according to claim 1. The information of the name of the GUI element is defined as attribute information related to the GUI element.
The operation recording apparatus according to claim 2. Event notification means for notifying the event detection means of occurrence of the predetermined type of event,
The event notification means notifies an event occurring in the one or more GUI elements having a tree structure to a GUI element of a root node of the one or more GUI elements.
The operation recording device according to any one of claims 1 to 3. The event notified by the event notification means is a tunnel event in WPF.
The operation recording apparatus according to claim 4. The event specifying means, when the type of the GUI element where the event detected by the event detection means is generated is included in a list of GUI element types prepared in advance, To identify,
The operation recording device according to any one of claims 1 to 5. The event information includes a value of attribute information related to the GUI element.
The operation recording device according to any one of claims 1 to 6. An operation recording method for recording a user operation in a GUI application including one or more GUI elements,
An event detection step of detecting a predetermined type of events detected in the GUI element;
An event identification step of identifying an event related to the user's operation from the detected event using information on a type of GUI element;
An event information recording step for recording information of the identified event;
An operation recording method. An operation recording program for causing a computer to execute an operation recording method for recording a user operation in a GUI application including one or more GUI elements,
An event detection step of detecting a predetermined type of events detected in the GUI element;
An event identification step of identifying an event related to the user's operation from the detected event using information on a type of GUI element;
An event information recording step for recording information of the identified event;
An operation recording program that causes a computer to execute.

Images