WO2022180863A1

WO2022180863A1 - User operation recording device and user operation recording method

Info

Publication number: WO2022180863A1
Application number: PCT/JP2021/007580
Authority: WO
Inventors: 史拓横瀬; 公雄土川; 佐也香八木; 有記卜部; 美沙深井; 晴夫大石
Original assignee: 日本電信電話株式会社
Priority date: 2021-02-26
Filing date: 2021-02-26
Publication date: 2022-09-01

Abstract

This user operation recording device (10) detects a user operation, selects an estimation method for estimating a delay time from a time when the user operation is detected to a time when a screen changes due to the user operation, depending on the type of application to be executed, and estimates the delay time by using the estimation method. Then, the user operation recording device (10) captures the screen operated by the user, on the basis of the estimated delay time, and acquires image data.

Description

User operation recording device and user operation recording method

The present invention relates to a user operation recording device and a user operation recording method.

Software that records user operations on a PC for purposes such as business analysis is known (see Non-Patent Document 1). In such software, the time, user name, application name, etc., as well as screen captures at the time of operation are recorded together and used for analysis of operation status.

However, in the conventional technology, since the screen is captured immediately after detecting the user's operation, it is possible to properly capture the screen during the operation in situations such as when it takes time for the screen to change after the user performs an operation. can be difficult to obtain.

An example of a case in which there is a large delay from when the user operates an input device such as a mouse or keyboard until a screen change such as a change on the GUI occurs due to that operation is, for example, "The processing load on the PC is increasing. "When the operated application is performing heavy processing other than GUI processing", "When communication delay affects thin client or remote desktop (hereinafter collectively referred to as remote desktop)", etc. .

In such cases, the time from when the mouse/keyboard is operated until the GUI display changes (response delay time) depends on the performance of the terminal, the execution status of the application, and in remote desktop, network delays, etc. The amount of delay varies greatly depending on the communication quality. In the past, this timing was adjusted manually, but since it was not possible to follow changes in the situation, it was not possible to properly acquire screen captures during operation.

The present invention has been made in view of the above, and it is an object of the present invention to provide a user operation recording device and a user operation recording method that can acquire screen captures at appropriate timing.

In order to solve the above-described problems and achieve the object, the user operation recording device of the present invention includes a detection unit that detects a user operation on an information processing device, Selecting an estimation method for estimating a delay time from when the user operation is detected by the detection unit until the screen of the information processing device changes due to the user operation, and estimating the delay time using the estimation method. An estimating unit, and an acquiring unit that acquires image data by capturing the screen based on the delay time estimated by the estimating unit.

Further, a user operation recording method of the present invention is a user operation recording method executed by a user operation recording device, comprising: a detection step of detecting a user operation on an information processing device; Accordingly, an estimation method for estimating a delay time from when the user operation is detected by the detection step until the screen of the information processing device changes due to the user operation is selected, and the delay time is estimated using the estimation method. and an acquisition step of acquiring image data by capturing the screen based on the delay time estimated by the estimation step.

According to the present invention, it is possible to acquire screen captures at appropriate timing.

FIG. 1 is a diagram illustrating the configuration of a system including a user operation recording device of this embodiment. FIG. 2 is a block diagram illustrating a schematic configuration of the user operation recording device of this embodiment. FIG. 3 is a diagram illustrating an example of a table stored in an estimation method storage unit; FIG. 4 is a diagram for explaining estimation processing by an estimation unit. FIG. 5 is a diagram for explaining estimation processing by the estimation unit. FIG. 6 is a diagram for explaining estimation processing by the estimation unit. FIG. 7 is a diagram for explaining statistical processing by an estimating unit; FIG. 8 is a flowchart illustrating an example of a processing procedure of user operation recording processing. FIG. 9 is a diagram illustrating an example of a computer that executes a user operation recording program;

Embodiments of a user operation recording device and a user operation recording method according to the present application will be described in detail below with reference to the drawings. Moreover, the present invention is not limited to the embodiments described below.

[System configuration including user operation recording device]
FIG. 1 is a diagram illustrating the configuration of a system including a user operation recording device of this embodiment. As shown in FIG. 1, a system including a user operation recording device 10 has a user operation recording device 10 and a server 20, and the user operation recording device 10 and the server 20 are connected via a network. In the example of FIG. 1, a case in which a local terminal (user operation recording device 10) at home or the like is remotely connected to the in-house server 20 to perform work will be described as an example, but this is limited to a remote environment. not a thing Note that the configuration shown in FIG. 1 is merely an example, and the specific configuration and the number of each device are not particularly limited.

The user operation recording device 10 is an information processing device (user terminal) used by the user. Note that the terminal device 200 may be any type of information processing device including client devices such as smart phones, desktop PCs, notebook PCs, and tablet PCs. In the example of FIG. 1, the user operation recording device 10, which is a user terminal, has a function for performing user operation recording processing. You may have

In the user operation recording device 10, a function of transmitting information input by a user's input device such as a mouse and keyboard to the server 20 and displaying screen display information received from the server 20 on the screen is in operation. This function is generally implemented as an application within the user terminal, and is displayed as a window like other applications within the user terminal. The screen display information received from the server 20 may be the entire virtual desktop on the server 20 side, or may be a window unit within the virtual desktop.

When the server 20 receives the operation information of the mouse, keyboard, etc., it performs an operation corresponding to the operation information to the application actually used by the user, and records the screen display information generated on the virtual desktop in the server as a user operation record. Send to device 10 .

With such a remote desktop, there may be a large response delay compared to applications executed within the user terminal. For example, operation delays may occur due to processing delays in communication paths, virtual servers, gateways, and the like. Therefore, the user operation recording device 10 estimates the response delay time of the GUI operation, and acquires GUI information such as screen capture at the optimum timing using the estimated response delay time.

[Configuration of user operation recording device]
FIG. 2 is a block diagram showing an example of the configuration of the user operation recording device. FIG. 2 is a block diagram illustrating a schematic configuration of the user operation recording device of this embodiment. As illustrated in FIG. 2, the user operation recording device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15. Prepare.

The input unit 11 is implemented using an input device such as a keyboard and a mouse, and inputs various instruction information such as processing start to the control unit 15 in response to input operations by the operator. The output unit 12 is implemented by a display device such as a liquid crystal display, a printing device such as a printer, or the like. For example, the output unit 12 is a screen on which a user operation is reflected, which is a target of user operation recording processing to be described later.

The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between an external device and the control unit 15 via an electrical communication line such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device or the like that manages various types of information regarding a user operation detection method, application processing, and the like. Note that the screen on which the user's operation is reflected may be implemented in hardware different from the user's operation recording device 10 . In that case, the user operation recording device 10 communicates with the terminal operated by the user via the communication control unit 13 .

The storage unit 14 stores data and programs necessary for various processes by the control unit 15, and has an estimation method storage unit 14a, an image data storage unit 14b, and a log storage unit 14c. For example, the storage unit 14 is a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk.

The estimation method storage unit 14a stores an estimation method for estimating the delay time from when a user operation is detected until the screen changes due to the user operation. For example, the estimation method storage unit 14a stores a table indicating which estimation method is to be applied for each window application.

Here, the table stored in the estimation method storage unit will be described using the example of FIG. FIG. 3 is a diagram illustrating an example of a table stored in an estimation method storage unit; As exemplified in FIG. 3, the estimation method storage unit 14a stores a "serial number", a "process name" that is the name of an application (process) displaying a window, and a "window title" that is the title of the window. and the "detection method type" indicating the type of delay time estimation method to be applied are stored in association with each other. In other words, since the optimal delay detection method differs depending on the application of the window, the table data shown in FIG. 3 is prepared so that the estimation unit 15b, which will be described later, can select an estimation method. Note that this method is the simplest method, and other more sophisticated methods may be used.

Also, regarding the method of estimating the delay time, the detection method (1) here is a method of estimating the delay time by measuring the processing time of the GUI event. The detection method (2) is a method of estimating the delay time by reading changes in the screen display caused by a dedicated application executed on the remote desktop server 20 side. The detection method (3) is a method of estimating the delay time by reading changes in the screen display at the clicked position.

For example, the process name "RemoteDesktopClient.exe" illustrated in FIG. 3 identifies the window displayed by the remote desktop application, and the screen information generated on the remote server 20 is displayed on the window. is assumed to be displayed. Also, as shown in FIG. 3, in the detection method (2), since it is assumed that a dedicated application is running on the remote server, the IP address of the connection destination is determined by the window title. . It is assumed that the address of the connection destination is displayed in the window title.

Also, "*" in the table means a wild card. For this reason, for example, in sequence number 3 in the example of FIG. 3, both the process name and window title match all patterns, so all windows that do not match the first and second lines are detected by detection method (1). is determined to be optimal. In the example of FIG. 3, one of the detection methods (1) to (3) is always used, but there may be windows that do not use the detection method. In that case, the response delay time is always regarded as constant. A constant value is either no delay (0 ms) or is given by some setting file or the like.

The image data storage unit 14b stores image data acquired by screen capture by the acquisition unit 15c, which will be described later. The log storage unit 14c stores log information in which an operation log including details of user operations and the like is associated with image data in which changes due to the user operations are reflected.

The control unit 15 has an internal memory for storing programs defining various processing procedures and required data, and executes various processing using these. For example, the control unit 15 has a detection unit 15a, an estimation unit 15b, an acquisition unit 15c, and a log creation unit 15d. Here, the control unit 15 is an electronic circuit such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The detection unit 15a detects user operations on the user operation recording device 10 (information processing device). Specifically, the detection unit 15a detects the user's operation using various detection methods. Further, the detection unit 15a assigns an ID or the like for individually identifying user operations detected among the processing units.

For example, the detection unit 15a uses the OS function to detect mouse and keyboard operations. This method of detecting mouse and keyboard operations is generally a high-speed method with very little delay, but it is not possible to determine how the OS and applications are affected by the operation.

Alternatively, the detection unit 15a monitors GUI messages, which are exchanges between a GUI system having a window mechanism and an application, to detect user operations. For example, the detection unit 15a detects appearance, disappearance, frontmost (activation), size change, and the like of a window. The detection unit 15a also detects that a button, text box, or the like for standard GUI control in the GUI system has been clicked.

Alternatively, the detection unit 15a detects a user operation, for example, using an API uniquely provided by the application. For example, the detection unit 15a can generally detect a button click or the like that occurs in a web page displayed on a web browser that has an API capable of detecting user operations. This detection method generally has a large delay.

Alternatively, the detection unit 15a detects a user operation by monitoring the content of communication between an application and a server or the like via a network, for example. Note that the operation detection method executed by the detection unit 15a is not limited to the above, and other more advanced operation detection methods may be used.

The estimating unit 15b detects the user operation by the detecting unit 15a according to the type of application executed by the user operation recording device 10 (information processing device), and detects the user operation by the user operation recording device 10 (information processing device). select an estimation method for estimating the delay time until the screen changes, and use the estimation method to estimate the delay time.

To explain using the example of FIG. 3, for example, the estimation unit 15b refers to the table stored in the estimation method storage unit 14a, the process name of the application (process) displaying the window, and the window of that window. Use the title to identify the estimation method. These two pieces of information are available in many OSs, and can be easily obtained by calling APIs provided in the OS. In the example of the table shown in FIG. 3, the estimating unit 15b performs comparisons in order from the row with the lowest serial number. do. Delay estimation processing by the detection methods (1) to (3) will be described below with reference to FIGS. 4 to 6, respectively. 4 to 6 are diagrams for explaining estimation processing by the estimation unit.

First, the detection method (1) in FIG. 3 will be described. For example, as one of estimation methods, the estimation unit 15b generates a GUI event for a window on the screen, measures the time from generation of the GUI event to completion of processing of the GUI event, This time is estimated as the delay time.

In the GUI system provided with the OS, the application has an event loop (also called a message loop) that processes GUI events. Depending on the type of OS, the event loop is processed on an application basis, a window basis, a thread basis, or the like, but in this embodiment, it is assumed that there is an event loop for each window. Even in the case of a GUI system in which event loops are prepared for each application and each thread, it is often considered that events are processed for each window.

Mouse and keyboard inputs are also notified to each window as GUI events, and the GUI display on the screen is updated in response. The event loop is usually single-threaded, and if response delays occur, it also affects the processing of events other than mouse and keyboard events. By using the OS API, it is possible to generate a GUI event (message) from another process (application) to a window on the desktop.

The estimating unit 15b uses this to measure the time from the generation of some GUI event to the completion of processing of the event for the window whose response delay is to be checked, and sets it as the response delay time. Using the example of FIG. 4, the estimating unit 15b causes the response delay estimation process to transmit an event to the target process (application) using the API of the OS, thereby generating a GUI event. to the reception of the event processing result indicating that the processing of the event has been completed is measured, and the measured time is estimated as the delay time.

Here, it is optimal that the GUI event to be generated does not change the counterpart application. take advantage of it. In a GUI system in which such an event does not exist, an event with less influence may be selected and used. For example, the estimation unit 15b applies the detection method (1) when an application other than a remote desktop application is the target application.

Next, the detection method (2) in FIG. 3 will be explained. The estimating unit 15b, for example, as one of the estimation methods, is a server 20 that executes an operation on a user's screen on a virtual desktop and transmits information about a screen display as an execution result. For the server 20 that executes an application that changes information related to screen display when an operation is detected, the detection unit 15a measures the time from when the user operation is detected until the screen changes, and estimates the measured time as the delay time. do.

In this case, an application that monitors and detects mouse and keyboard operation information received by the server 20 side, encodes the detected operation information each time it detects it, and reflects it in part of the screen display information sent from the server 20. is executed on the desktop of the server 20. Operation information received by the server 20 can be monitored by the same method as the detection unit 15 a of the user operation recording device 10 . As illustrated in FIG. 5, the image displayed on the screen uses, for example, a two-dimensional code. Since the two-dimensional code takes up an area and stands out, in order not to hide the display to be seen by the user, a color bar code using only the pixels at the edges of the screen display information transmitted from the server 20, or an electronic watermark is used. You may The estimation unit 15b updates this image each time it receives operation information from the server 20. FIG. The two-dimensional code includes coordinate information, button type and up/down information for a mouse, and key type and up/down information for a keyboard.

It should be noted that the two-dimensional code may contain not only information about the latest operation, but also information about multiple operations as an operation history. In other words, when the user performs a plurality of operations in succession in a short period of time, the two-dimensional code may change frequently and the two-dimensional code may not be properly read. In such a case, for example, the two-dimensional code includes information about a plurality of operations as an operation history, and the sequence number attached to each operation and the update of the two-dimensional code after the server 20 detects the operation. It may also include information about the time difference between

The program for detecting the delay time in the user operation recording device 10 detects mouse and keyboard operation information and accumulates the history along with the detection time. Furthermore, the user operation recording device 10 periodically monitors the display of the two-dimensional code on the remote desktop application. Then, the user operation recording device 10 detects the operation information of the mouse and the keyboard received by the server 20 from the information included in the two-dimensional code, compares it with the history of the operation information generated in the user operation recording device 10, Estimate the delay. That is, the user operation recording device 10 compares the detection time of each operation accumulated as a history with the time when a change in display of the two-dimensional code is detected for the same operation, calculates the difference between the two, and calculates the difference. is estimated as the delay time.

Also, the user operation recording device 10 needs to monitor changes in the image on the remote desktop application at sufficiently short intervals. Since the display area of the two-dimensional code or the like to be monitored by the user operation recording device 10 is sufficiently small, it can be realized with a small processing load. According to the mechanism of this detection method (2), the response from when the user operates the mouse or keyboard, when the information is sent to the server 20, and is actually reflected on the screen display of the user operation recording device 10 Delay can be estimated.

Next, the detection method (3) in FIG. 3 will be explained. For example, as one of the estimation methods, the estimating unit 15b monitors a change in an area within a predetermined range from the click position where the click operation is performed when the detecting unit 15a detects a click operation as a user operation. , the time from the detection of the click operation by the detection unit 15a to the occurrence of a change in the area is measured, and the measured time is estimated as the delay time.

With detection method (2), it is necessary to run a dedicated application in the remote desktop server 20, and it may not be possible in all environments. Therefore, the detection method (3) is a method capable of estimating the delay time without adding a special function to the server 20 side. Detection method (3) uses only click information. When a click operation is performed, in many cases, the property that the display changes at the clicked position is utilized.

When the user operation recording device 10 detects a click operation to the remote desktop application within its own device, it monitors a small area centered on the clicked position, and monitors the image at the moment of the click until the change occurs. Measure the time and use it as the response delay time. The user operation recording device 10 needs to monitor changes in the image on the remote desktop application at sufficiently short intervals, but since the area to be monitored is sufficiently small, it can be implemented with a small processing load.

Note that the user operation recording device 10 may perform statistical processing to estimate more accurate values because there is a possibility that an image will not change due to a click or that an image will change regardless of the click. can be Statistical processing may be performed not only for the detection method (3) but also for the detection methods (1) and (2). In other words, in any detection method, a more appropriate value can be estimated by statistical processing because variations occur in single measurements.

For example, the estimation unit 15b may aggregate a plurality of delay times estimated by the same estimation method for each predetermined time range, and estimate the median value of the time range with the highest frequency as the delay time. . Here, statistical processing will be described using the example of FIG. FIG. 7 is a diagram for explaining statistical processing by an estimating unit;

For example, the estimating unit 15b subjects data within a certain time range from the current time to the past for statistical processing. For example, the measurement results for the past 10 minutes are statistically processed to obtain the current response delay time. Then, the estimating unit 15b removes a single delay time measurement value that is equal to or greater than a threshold value. For example, measured values of 3 seconds or longer are removed as singular values. This is the same even if the measurement is discontinued when the threshold value is exceeded in the single measurement process.

Subsequently, the estimation unit 15b aggregates the remaining measured values for each time range. For example, the measured values are aggregated with a time width of 0.2 seconds. Then, as illustrated in FIG. 7, the estimation unit 15b estimates that the true value of the response delay time is included in the time range with the highest frequency, and adopts the median value of that time range as the delay time. Note that the estimation unit 15b is not limited to the above processing, and may perform more advanced statistical processing.

For example, in the detection method (3), the estimating unit 15b may not affect the screen display by clicking, but in this case, it is removed by the threshold. In addition, changes in the screen display that are not caused by mouse or keyboard operations may be erroneously detected, but since the changes vary widely over time, the effects can be eliminated by selecting a time range with a high frequency. The estimating unit 15b may store the delay time for each application in the storage unit 14 after performing statistical processing to estimate the delay time.

Returning to the description of FIG. 2, the acquisition unit 15c acquires image data by capturing the screen based on the delay time estimated by the estimation unit 15b. For example, the acquiring unit 15c acquires the image data of the screen after the delay time estimated by the estimating unit 15b has elapsed since the detection unit 15a detected the user's operation. Then, the acquisition unit 15c stores the acquired image data in the image data storage unit 14b.

In addition, the acquisition unit 15c may store the delay time for each application and acquire image data by capturing screens operated by the user at predetermined intervals. In this case, the acquiring unit 15c selects image data of the screen after the delay time estimated by the estimating unit 15b has elapsed since the detecting unit 15a detected the user's operation from the acquired image data. and stored in the image data storage unit 14b. Further, when the delay time for each application is stored in the storage unit 14, the obtaining unit 15c reads the delay time according to the target application whose window is being displayed, and Image data may be acquired and stored in the image data storage unit 14b.

The log creation unit 15d associates the selected image data with the operation log and outputs them. For example, the log creation unit 15d stores, as a log, a text file corresponding to each line of each detected operation in the log storage unit 14c of the storage unit 14, and also creates each image corresponding to each detected operation. The data is stored in the log storage unit 14c with identification information added to the file name or the like so that the corresponding relationship between each line of the text file and each image file can be known as each image file. Alternatively, the log creating unit 15e may output the operation log and the image data as single binary data. Further, instead of storing the log in the log storage unit 14c of the storage unit 14, the log creation unit 15e may transmit to another server or the like via the communication control unit 13. FIG.

[Processing procedure of user operation recording device]
Next, an example of a processing procedure of user operation recording processing executed by the user operation recording device 10 will be described with reference to FIG. 8 . FIG. 8 is a flowchart illustrating an example of a processing procedure of user operation recording processing.

As illustrated in FIG. 8, when the detection unit 15a of the user operation recording device 10 detects a user operation on the screen (Yes at step S101), the estimation unit 15b selects the detection unit 15a according to the application type for displaying the screen. selects an estimation method for estimating a delay time from the detection of a user operation until the screen changes due to the user operation (step S102).

Then, the estimation unit 15b estimates the delay time using an estimation method (step S103). Subsequently, the acquiring unit 15c acquires image data by capturing the screen operated by the user based on the delay time estimated by the estimating unit 15b (step S104).

After that, the log creating unit 15d associates and records the selected image data and the operation log (step S105). For example, the log creation unit 15d stores, as a log, a text file corresponding to each line of each detected operation in the log storage unit 14c of the storage unit 14, and also creates each image corresponding to each detected operation. The data is stored in the log storage unit 14c with identification information added to the file name or the like so that the corresponding relationship between each line of the text file and each image file can be known as each image file.

[Effects of Embodiment]
As described above, the user operation recording apparatus 10 according to the embodiment detects a user operation, and according to the type of application for displaying a screen, the delay time from detection of the user operation to change of the screen due to the user operation is determined. is selected, and the delay time is estimated using the estimation method. Based on the estimated delay time, the user operation recording device 10 acquires image data by capturing the screen operated by the user. Therefore, the user operation recording device 10 can acquire screen captures at appropriate timing.

In other words, screen captures are not limited to looking back in the past, but there are cases where it is necessary to acquire an image in the future (after waiting for a certain amount of time) after an operation is detected. In addition to screen capture, when acquiring various types of GUI information (window title, etc.), it is necessary to acquire the information after the operation is reflected. It is necessary to wait for a certain period of time, for example, when there is a large delay between detecting a mouse/keyboard operation and causing a change on the GUI due to the operation. Even in such a case, the user operation recording device 10 can estimate the response delay time of the GUI operation according to the state change of the computer or network at that time. By using this estimated delay time, acquisition of various GUI information including screen capture can be performed at optimum timing.

For example, in the user operation recording device 10, depending on the type of application, "a method of measuring the processing time of a GUI event", "a method of installing a dedicated application on the server 20 side and reading the response from an image", and "screen display of the clicked position". It is possible to acquire screen captures at appropriate timing by switching to the optimum detection method from among the "methods for reading changes in" and quantitatively estimating the response delay time of the GUI.

Note that in the user operation recording device 10, the process of obtaining the delay time of each window may be always performed in parallel for all windows on the desktop, but basically input from the mouse and keyboard is accepted. Since many OSs have only one window (called an active window or the like), it is often sufficient to process only the active window. Therefore, in the present embodiment, processing may be performed only for the active window.

However, in this case, since the active window is generally switched frequently, the user operation recording device 10 may retain information on the delay of windows other than the active window in a memory or the like. For example, in the user operation recording device 10, at the timing when a window is newly displayed, an area for holding information about the window is secured, a response delay is obtained while the window is active, and the information in the held area is rewritten. . When the window is destroyed, its area may be destroyed accordingly.

[About the system configuration of the embodiment]
Each component of the user operation recording device 10 shown in FIG. 1 is functionally conceptual, and does not necessarily need to be physically configured as shown. That is, the specific form of distributing and integrating the functions of the user operation recording device 10 is not limited to the illustrated one, and all or part of it can be functionally or It can be physically distributed or integrated.

Also, each process performed in the user operation recording device 10 may be realized entirely or in part by a CPU and a program that is analyzed and executed by the CPU. Further, each process performed in the user operation recording device 10 may be implemented as hardware based on wired logic.

Also, among the processes described in the embodiments, all or part of the processes described as being performed automatically can also be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the above-described and illustrated processing procedures, control procedures, specific names, and information including various data and parameters can be changed as appropriate unless otherwise specified.

[program]
It is also possible to create a program in which the processing executed by the user operation recording device 10 according to the above embodiment is described in a computer-executable language. As one embodiment, the user operation recording device 10 can be implemented by installing a user operation recording program for executing the user operation recording process as package software or online software in a desired computer. For example, the information processing device can function as the user operation recording device 10 by causing the information processing device to execute the user operation recording program. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, information processing devices include smart phones, mobile communication terminals such as mobile phones and PHSs (Personal Handyphone Systems), and slate terminals such as PDAs (Personal Digital Assistants). Also, the functions of the user operation recording device 10 may be implemented in a cloud server.

FIG. 9 is a diagram showing an example of a computer that executes a user operation recording program. Computer 1000 includes, for example, memory 1010 , CPU 1020 , hard disk drive interface 1030 , disk drive interface 1040 , serial port interface 1050 , video adapter 1060 and network interface 1070 . These units are connected by a bus 1080 .

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 . The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1031 . Disk drive interface 1040 is connected to disk drive 1041 . A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041, for example. A mouse 1051 and a keyboard 1052 are connected to the serial port interface 1050, for example. For example, a display 1061 is connected to the video adapter 1060 .

Here, the hard disk drive 1031 stores an OS 1091, application programs 1092, program modules 1093 and program data 1094, for example. Each piece of information described in the above embodiment is stored in the hard disk drive 1031 or the memory 1010, for example.

Also, the user operation recording program is stored in the hard disk drive 1031 as a program module 1093 in which commands to be executed by the computer 1000 are described, for example. Specifically, the hard disk drive 1031 stores a program module 1093 that describes each process executed by the user operation recording apparatus 10 described in the above embodiment.

Data used for information processing by the user operation recording program is stored as program data 1094 in the hard disk drive 1031, for example. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the hard disk drive 1031 to the RAM 1012 as necessary, and executes each procedure described above.

Note that the program modules 1093 and program data 1094 related to the user operation recording program are not limited to being stored in the hard disk drive 1031. For example, they may be stored in a removable storage medium and processed by the CPU 1020 via the disk drive 1041 or the like. may be read out. Alternatively, the program module 1093 and program data 1094 related to the user operation recording program are stored in another computer connected via a network such as a LAN or WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070. may be issued.

Although the embodiment to which the invention made by the present inventor is applied has been described above, the present invention is not limited by the descriptions and drawings forming part of the disclosure of the present invention according to the present embodiment. That is, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on this embodiment are all included in the scope of the present invention.

10 user operation recording device 11 input unit 12 output unit 13 communication control unit 14 storage unit 14a estimation method storage unit 14b image data storage unit 14c log storage unit 15 control unit 15a detection unit

15b estimation unit

15c acquisition unit 15d log creation unit

Claims

a detection unit that detects a user operation on the information processing device;
selecting an estimation method for estimating a delay time from when a user operation is detected by the detection unit to when the screen of the information processing device changes due to the user operation, according to the type of application executed by the information processing device; , an estimation unit that estimates the delay time using the estimation method;
an acquisition unit that acquires image data by capturing the screen based on the delay time estimated by the estimation unit;
A user operation recording device comprising:
The estimating unit, as one of the estimating methods, is a server that executes an operation on a user's screen on a virtual desktop and transmits information about a screen display as an execution result. for a server that executes an application that changes information related to the screen display when is detected, measuring the time from when the user operation is detected by the detection unit until the screen changes, and the measured time is defined as the delay time. 2. The user operation recording device according to claim 1, wherein the estimation is performed.
As one of the estimation methods, the estimating unit monitors a change in an area within a predetermined range from a click position where the click operation is performed when the detecting unit detects a click operation as the user operation. 2. The user operation according to claim 1, wherein the time from when the click operation is detected by the detection unit to when the change occurs in the area is measured, and the measured time is estimated as the delay time. recording device.
As one of the estimation methods, the estimation unit generates a GUI event for a window on the screen, measures the time from generation of the GUI event to completion of processing of the GUI event, and 2. The user operation recording device according to claim 1, wherein the delay time is estimated as the delay time.
The estimation unit aggregates a plurality of delay times estimated by the same estimation method for each predetermined time range, and estimates the median value of the delay times included in the time range with the largest total value as the delay time. The user operation recording device according to claim 1, characterized by:
A user operation recording method executed by a user operation recording device,
a detection step of detecting a user operation on the information processing device;
selecting an estimation method for estimating a delay time from when a user operation is detected by the detection step to when the screen of the information processing device changes due to the user operation, according to the type of application executed by the information processing device; , an estimation step of estimating the delay time using the estimation method;
an acquisition step of acquiring image data by capturing the screen based on the delay time estimated by the estimation step;
A user operation recording method, comprising: