WO2022034672A1

WO2022034672A1 - Data processing device and data processing method

Info

Publication number: WO2022034672A1
Application number: PCT/JP2020/030801
Authority: WO
Inventors: 有記卜部; 公雄土川; 晴夫大石; 史拓横瀬; 佐也香八木
Original assignee: 日本電信電話株式会社
Priority date: 2020-08-13
Filing date: 2020-08-13
Publication date: 2022-02-17
Also published as: JP7439934B2; JPWO2022034672A1

Abstract

The present invention provides a data processing device that receives setting information indicating an event to be detected from among events that occur in a terminal device from which a remote connection originates. Thereafter, if the data processing device determines from the event to be detected that a change has occurred in a screen of the terminal device on the basis of the setting information and a screen capture from the terminal device acquired at predetermined time intervals, the data processing device outputs, as an operation delay time, the value of the difference between the time at which the event occurred and the time of acquiring the screen capture in which the change occurred.

Description

Data processing equipment and data processing method

The present invention relates to a data processing apparatus and a data processing method.

In order to improve work styles and prevent infectious diseases, work may be carried out by remotely connecting from a local terminal such as home to an in-house terminal or virtual terminal. In the remote connection as described above, operation delay may occur due to communication delay, processing delay of virtual server, gateway, etc., which may lead to an increase in stress of the operator and a decrease in productivity.

Here, in order to detect the occurrence of operation delay, a method of measuring the load on an in-house server or network (see Non-Patent Document 1), or a method of measuring the time taken for operation on a screen accessible from a terminal. (See Non-Patent Document 2).

However, the method of measuring the load on the server or network in the company cannot grasp the processing delay experienced by the worker. Further, the method of measuring the time required for the operation on the screen accessible from the terminal requires time and effort for the operator, and has a problem in the reproducibility of the operation.

Therefore, it is an object of the present invention to solve the above-mentioned problem and automatically measure the operation delay time experienced by the operator in the remote connection.

In order to solve the above-mentioned problems, the present invention has a setting unit that receives setting information indicating an event to be detected among the events that occur in the connection source terminal of the remote connection, the setting information, and every predetermined time. When it is determined that a change in the screen of the connection source terminal has occurred due to the event to be detected based on the acquired capture of the screen of the connection source terminal, the occurrence time of the event and the change are determined. It is characterized by including an output processing unit that outputs the value of the difference from the acquisition time of the generated screen capture as the operation delay time.

According to the present invention, it is possible to automatically measure the operation delay time experienced by the operator in the remote connection.

FIG. 1 is a diagram for explaining an outline of operation of a data processing device in the system of the first embodiment. FIG. 2 is a diagram showing a configuration example of the system of the first embodiment. FIG. 3 is a flowchart showing an example of the processing procedures of processing 1 and processing 2 performed by the system shown in FIG. FIG. 4 is a flowchart showing an example of the processing procedure of the processing 3 performed by the system shown in FIG. FIG. 5A is a diagram for explaining an outline of operation of the data processing device in the system of the second embodiment. FIG. 5B is a flowchart showing an example of the processing procedure of processing 3 performed by the system of the second embodiment. FIG. 6 is a diagram showing a configuration example of the system of the third embodiment. FIG. 7 is a flowchart showing an example of the processing procedure of processing 3 performed by the system of the third embodiment. FIG. 8 is a flowchart showing an example of the processing procedure of processing 4 performed by the system of the second embodiment. FIG. 9 is a diagram showing a configuration example of a computer that executes a data processing program.

Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to each embodiment.

[First Embodiment]
[Overview]
The outline of the operation of the data processing apparatus in the system of the first embodiment will be described with reference to FIG.

In the following description, the terminal device is a local terminal (connection source terminal) that connects to another terminal device by remote connection. For example, a terminal device is connected to another terminal device via a VDI (Virtual Desktop Infrastructure) server. The data processing device sets the processing time of EtoE (from the operation issued from the local terminal to the screen change on the local terminal) when the above terminal device remotely connects to another terminal device as the operation time of the local device. It is calculated as the operation delay time experienced by the person.

Here, the data processing device receives in advance the setting information of the operation event (event to be detected) of the terminal device used for calculating the above-mentioned EtoE processing time. The setting information is described by, for example, a combination of an operation event to be detected and an image of a portion that changes on the screen of the terminal device due to the operation event.

Here, for example, a case where the key event "A" and the images "A" and "A" shown in FIG. 1 are set in the setting information as the event to be detected will be described as an example. Since the appearance (for example, display size, font, color, etc.) of the image corresponding to the key event changes depending on the application used on the terminal device 10, it is preferable to set various possible images.

<Process 1>
For example, the data processing device stores an operation event (input event) of the terminal device in the input event database. For example, when "A" is input to the terminal device, the data processing device stores the input event of "A" and the occurrence time of the input event (event) in the input event database in association with each other.

<Process 2>
Further, for example, the data processing device acquires a screen capture of the terminal device at predetermined time intervals, and acquires a change portion by comparing the screen captures before and after. Then, the data processing device stores the changed part of the acquired screen capture and the acquisition time of the screen capture in the screen capture database. For example, the data processing device stores the screen capture, the change point of the screen capture (for example, the coordinate area of the change point "A"), and the acquisition time of the screen capture in the screen capture database.

<Process 3>
After that, the data processing device acquires the input event to be detected (for example, the input event of "A") from the input event database. Further, the data processing device acquires a screen capture (screen capture of the detection target) corresponding to the input event of the detection target from the screen capture database. For example, in the data processing device, from the screen capture database, the time when the screen capture occurs is after the time when the input event to be detected occurs, and the image of the changed part in the screen capture is "A" or "A". Get the screen capture data that is. As a method for the data processing device to determine whether the screen capture data in the screen capture database is a screen capture including a change portion corresponding to the input event of the detection target, for example, an image set in the setting information and the change portion. If it is considered that both the size and the font match in the image of, the data processing device converts each of the image set in the setting information and the image of the changed part into a gray scale image, further binarizes, and then the template. The similarity is calculated using matching. Then, if the calculated similarity is equal to or higher than the set threshold value, the data processing device may determine the screen capture corresponding to the event to be detected. In addition, when it is considered that the fonts of the image set in the setting information and the image of the change part are different but the fonts match, the data processing device grayscales both the image set in the setting information and the image of the change part. After scaling, features are extracted using SIFT (Scale-Invariant Feature Transform), and if the degree of matching of features is equal to or higher than the threshold value, it is judged as a screen capture corresponding to the input event to be detected. Then, the data processing device calculates the difference between the acquisition time of the acquired screen capture and the occurrence time of the input event. That is, the data processing device calculates the EtoE processing time by setting the acquisition time of the screen capture as the screen reflection time after the operation delay.

This allows the data processing device to automatically calculate the operation delay time experienced by the operator of the terminal device when the terminal device is remotely connected.

[Configuration example]
Next, a configuration example of a system including a data processing device will be described with reference to FIG. The system includes a terminal device 10 and a data processing device 20. The number of terminal devices 10 and data processing devices 20 is not limited to the number shown in FIG.

<Terminal device>
First, the terminal device 10 will be described. As described above, the terminal device 10 is a local device that connects to another terminal device by remote connection. The terminal device 10 includes an input acquisition unit 11, a screen acquisition unit 12, a screen comparison unit 13, an output unit 14, and a communication control unit 15.

The input acquisition unit 11 acquires an input event by an input device (for example, a keyboard, a mouse, etc.) of the terminal device 10.

The screen acquisition unit 12 acquires a screen capture of the terminal device 10 at predetermined time intervals. The screen comparison unit 13 acquires a change portion on the screen by using a screen capture acquired by the screen acquisition unit 12 at predetermined time intervals.

The output unit 14 outputs the EtoE processing time output from the data processing device 20. The communication control unit 15 controls communication between the terminal device 10 and the data processing device 20.

<Data processing device>
Next, the data processing device 20 will be described. The data processing device 20 includes a communication control unit 21, a control unit (output processing unit) 22, and a storage unit 23.

The communication control unit 21 controls communication between the data processing device 20 and the terminal device 10. The control unit 22 controls the entire data processing device 20. The control unit 22 includes a data acquisition unit 221, a determination unit 222, a setting unit 223, a search unit 224, and a measurement unit 225.

The data acquisition unit 221 acquires an input event in the terminal device 10 and a change point on the screen obtained from the screen capture of the terminal device 10 from the terminal device 10 via the communication control unit 21. Then, the data acquisition unit 221 stores the acquired input event data (input event data) in the input event database of the storage unit 23. The input event data is information indicating the input event and the occurrence time of the input event (see FIG. 1).

Further, the data acquisition unit 221 stores the acquired screen capture data including the changed part on the screen in the screen capture event database of the storage unit 23. This screen capture data is information indicating the screen capture, the change points on the screen obtained from the screen capture, and the acquisition time of the screen capture (see FIG. 1). The input event data and the screen capture data may be held in the memory of the data processing device 20, respectively.

The determination unit 222 determines whether or not the change portion of the screen obtained from the input event of the terminal device 10 and the image capture satisfies the content of the setting information set by the setting unit 223.

The setting unit 223 accepts the input of setting information. As described above, the setting information is described by, for example, a combination of an input event generated in the terminal device and an image (recognition target image) of a portion changed on the screen of the terminal device due to the input event. The setting information is input, for example, via an input unit (not shown) of the data processing device 20, and is stored in a predetermined area of the storage unit 23.

The search unit 224 accesses the input event database and the screen capture database, and searches for the screen capture corresponding to the input event.

The measurement unit 225 measures the value of the difference between the occurrence time of the input event searched by the search unit 224 and the acquisition time of the screen capture corresponding to the input event. Then, the measurement unit 225 outputs the measured difference value as the EtoE processing time.

The storage unit 23 includes an input event database and a screen capture database in a predetermined area. The input event database is a database that stores input events acquired by the data acquisition unit 221. The screen capture database is a database that stores the screen capture data including the changed part acquired by the data acquisition unit 221 in the screen capture database.

Further, the function of the data processing device 20 may be installed in the terminal device 10. In this case, the communication control unit 21 of the data processing device 20 and the communication control unit 21 of the terminal device 10 are unnecessary.

<Example of processing procedure>
Next, the processes (processes 1, 2, and 3) executed by the system will be described with reference to FIGS. 3 and 4. The process 1 is a process in which the system stores the input event acquired by the terminal device 10 in the input event database. The process 2 is a process in which the system stores the screen capture data acquired by the terminal device 10 in the screen capture database. The process 3 is a process in which the system obtains the EtoE processing time based on the input event data and the screen capture data obtained in the processes 1 and 2.

Process 1 and process 2 are processes performed in parallel in the background of the terminal device 10. Further, the process 3 is started, for example, at the timing instructed by the user of the data processing device 20. The setting unit 223 of the data processing device 20 shall accept the input of the setting information of the event to be detected in advance.

<Process 1>
First, the process 1 will be described with reference to FIG. The terminal device 10 acquires an input event of the terminal device 10 (S11). For example, the terminal device 10 acquires the key input in the terminal device 10 and the time when the input occurs as an input event. After that, when the data processing device 20 acquires the input event of the terminal device 10, the data processing device 20 stores the data (input event data) indicating the acquired input event in the input event database (S12).

After that, if the user of the terminal device 10 does not stop the process and the terminal device 10 is not shut down (No in S13), the process returns to S11. On the other hand, if the user of the terminal device 10 has stopped the process / the terminal device 10 has shut down (Yes in S13), the process is terminated.

<Process 2>
Subsequently, the process 2 will be described with reference to FIG. The terminal device 10 captures the screen of the terminal device 10 at predetermined time intervals (for example, every 50 ms). Then, the terminal device 10 compares the acquired screen captures, and if there is a change portion on the previous and next screens, acquires the change portion (S21). After that, when the data processing device 20 acquires the screen capture data including the change portion acquired in S21 from the terminal device 10, the acquired screen capture data is stored in the screen capture database (S22). After that, if the user of the terminal device 10 does not stop the process and the terminal device 10 has not shut down (No in S23), the process returns to S21. On the other hand, if the user of the terminal device 10 has stopped the process / the terminal device 10 has shut down (Yes in S23), the process is terminated.

<Process 3>
Next, the process 3 will be described. In the process 3, first, the data processing device 20 accesses the input event database and acquires the occurrence time of the input event to be detected. Next, the data processing device 20 accesses the screen capture database and searches for the screen capture data of the detection target that first occurs after the occurrence time of the input event of the detection target. After that, the data processing device 20 obtains a value of the difference between the occurrence time of the input event to be detected and the acquisition time of the screen capture in the searched screen capture data. Then, the data processing device 20 outputs the value of the difference as the processing time of EtoE.

Hereinafter, an example of the above process 3 will be described with reference to FIG. For example, the data processing device 20 accesses the input event database, selects the target input event data (S31), and determines whether or not the selected target input event data is an input event to be detected (S32). For example, the data processing device 20 selects one input event from the input event database, and determines whether or not the selected input event is an input event to be detected indicated in the setting information.

Next, access the data processing device 20 and the screen capture database, and select the target screen capture event (S33). For example, the data processing device 20 selects one screen capture data from the screen capture database.

After that, the data processing device 20 determines whether or not the target screen capture event selected in S33 is the screen capture data to be detected and occurs after the occurrence time of the target input event (S34). For example, in the data processing device 20, the image of the change portion in the screen capture data selected in S33 matches the image to be detected (recognition target image), and the acquisition time of the screen capture in the screen capture data is set. Judge whether it is after the occurrence time of the target input event.

Here, when the data processing device 20 determines that the target screen capture data selected in S33 is the screen capture data to be detected and occurs after the occurrence time of the target input event (Yes in S34). ), Calculate the value of the difference between the occurrence time of the target input event and the acquisition time of the target screen capture event (S35). Then, the data processing device 20 outputs the value of the difference calculated in S35 (S36). After that, when it is determined that the data processing device 20 has confirmed all the input events in the input event database (Yes in S38), the processing ends. On the other hand, when the data processing device 20 determines that any input event in the input event database is unconfirmed (No in S38), the data processing device 20 returns to S31.

The data processing device 20 determines that the target screen capture data selected in S33 is not the screen capture data to be detected, or that the target screen capture data has occurred before the time when the target input event occurs. In the case (No of S34), the process proceeds to S37. Then, when the data processing device 20 determines that all the screen capture data of the screen capture database has been confirmed (Yes in S37), the process proceeds to S38. On the other hand, when the data processing device 20 determines that any of the screen capture data in the screen capture database is unconfirmed (No in S37), the data processing device 20 returns to S33.

By doing so, the system can calculate the processing time of EtoE in the terminal device 10 to be remotely connected.

In the above example, the system has described the case where the determination of whether or not the input event and the screen capture data are the detection targets is performed in the process 3, but the present invention is not limited to this. For example, the system may determine whether or not the input event is the detection target in the process 1, and may determine whether or not the screen capture data is the detection target in the process 2. In this case, the system does not need to perform the determination process of whether or not the input event and the screen capture data are the detection targets in the process 3.

[Second Embodiment]
The system may calculate the above-mentioned EtoE processing time by using an event related to a change in the appearance of a button due to a mouse click of the terminal device 10. The embodiment in this case will be described as the second embodiment. The same configurations as those of the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

[Overview]
First, an outline of the operation of the data processing device in the system of the second embodiment will be described with reference to FIG. 5A.

In the second embodiment, an input event related to mouse operation is set as an event to be detected in the setting information. For example, a mouse click operation (mouse click) is set as an event to be detected in the setting information.

<Process 1>
Further, also in the system of the second embodiment, the above-mentioned processes 1, 2 and 3 are performed, but in the process 1, the terminal device 10 generates the mouse click and the mouse click in the terminal device 10 as input events. Acquires the coordinate position of the cursor (the place where the mouse click occurs), the time when the mouse click occurs, and the like. Then, the data processing device 20 stores the input event in the terminal device 10 in the input event database.

<Process 2>
Further, also in the process 2 of the system of the second embodiment, for example, the data processing device 20 acquires the screen capture of the terminal device 10 at predetermined time intervals, and acquires the changed portion by comparing the screen captures before and after. Then, the data processing device 20 stores the changed part of the acquired screen capture and the acquisition time of the screen capture in the screen capture database. For example, in the data processing device 20, when the save button on the screen is pressed by clicking the mouse of the terminal device 10 and the appearance of the save button changes, the screen capture at that time and the changed part of the screen capture (for example, save). The button coordinate area) and the acquisition time of the screen capture are stored in the screen capture database.

<Process 3>
Further, also in the process 3 in the system of the second embodiment, the data processing device 20 finds the input event to be detected shown in the setting information from the input event database. Then, the data processing device 20 acquires the place where the input event occurs and the time when the input event occurs.

Then, the data processing device 20 accesses the screen capture database and searches for the screen capture data in which the screen change occurs before and after the occurrence time of the input event to be detected. Then, the data processing device 20 determines whether or not the screen capture data corresponds to the input event to be detected. For example, the data processing device 20 determines whether or not the location where the input event (mouse click) to be detected is generated is included in the region of the change location in the searched screen capture data. Here, when the data processing device 20 determines that the location where the mouse click occurs is included in the region of the change portion in the searched screen capture data (for example, the coordinate region of the save button), the screen capture data is detected. It is determined that it corresponds to the input event of. Then, the data processing device 20 acquires the time when the screen capture occurs in the screen capture data. After that, the data processing device 20 calculates the value of the difference between the occurrence time of the input event and the acquisition time of the screen capture. Then, the data processing device 20 outputs the value of the difference as the processing time of EtoE.

In the second embodiment, for example, a mouse click operation (mouse click) and a cursor coordinate area in the click operation may be set in the setting information as an event to be detected.

In this case, in the process 3 of the system of the second embodiment, the data processing device 20 clicks in the coordinate area of the cursor indicated in the setting information as an input event to be detected indicated in the setting information from the input event database. Find input events related to the operation. Then, the data processing device 20 acquires the coordinate position of the cursor in the input event and the occurrence time of the input event. Subsequent processing is the same as the above-mentioned processing 3, so the description thereof will be omitted.

Hereinafter, an example of the process 3 in the system of the second embodiment will be described with reference to FIG. 5B. For example, the data processing device 20 accesses the input event database, selects the target input event data (S41), and determines whether or not the selected target input event data is an input event to be detected (S42). For example, the data processing device 20 selects one input event from the input event database, and determines whether or not the selected input event is an input event to be detected indicated in the setting information.

Next, access the data processing device 20 and the screen capture database, and acquire the screen captures that occurred before and after the target input event (S43). For example, the data processing device 20 acquires screen capture data from the screen capture database whose acquisition time of the screen capture is before or after the occurrence time of the target input event.

After that, the data processing device 20 determines whether or not the target input event is included in the change points of the screen capture before and after the target input event (S44). For example, the data processing device 20 determines whether or not the coordinate position of the cursor in the target input event is included in the area of the change portion in the screen capture data acquired in S43.

Here, when the data processing device 20 determines that the cursor coordinates of the target input event are included in the change points of the screen capture before and after the target input event (Yes in S44), the occurrence time of the target input event and the target screen capture. The difference from the acquisition time of the event is calculated (S45). The acquisition time of this target screen capture event is, for example, the acquisition time of the screen capture in the later screen capture data among the screen capture data determined in S44 that the target input event is included in the change location. .. Then, the data processing device 20 outputs the value of the difference calculated in S45 (S46). After that, when it is determined that the data processing device 20 has confirmed all the input events in the input event database (Yes in S47), the processing ends. On the other hand, when the data processing device 20 determines that any input event in the input event database is unconfirmed (No in S47), the process returns to S41.

If the target input event data selected in S41 by the data processing device 20 is not an input event to be detected (No in S42), the process proceeds to S47. Further, when the data processing device 20 determines in S43 that the cursor coordinate position of the target input event is not included in the change points of the screen capture before and after the target input event (No in S44), the process proceeds to S47.

By doing so, the system can calculate the EtoE processing time using the event related to the mouse operation of the terminal device 10.

In the above example, the system has described the case where it is determined in the process 3 whether or not the input event is the detection target, but the present invention is not limited to this. For example, the system may determine whether or not the input event is a detection target in the process 1. In this case, the system does not need to perform the determination process of whether or not the input event is the detection target in the process 3.

[Third Embodiment]
The system may calculate the EtoE processing time by using an event related to a change in the window state of the local terminal as an event to be detected. In this case, an event related to a change in the window state to be detected (for example, the active window is closed, etc.) is set in the setting information used by the system. The embodiment in this case will be described as a third embodiment. The same configurations as those of the above-described embodiments are designated by the same reference numerals, and the description thereof will be omitted.

[Configuration example]
A configuration example of the system of the third embodiment will be described with reference to FIG. The system includes a terminal device 10, a terminal device 10a, and a data processing device 20a.

The terminal device 10a is a terminal device (connection destination terminal) that is remotely connected from the terminal device 10 and provides a remote environment to the terminal device 10. The terminal device 10a includes a window state acquisition unit 16. The window state acquisition unit 16 acquires information on the window and process of the terminal device 10 in the remote environment at predetermined time intervals. Then, if there is a change in the window or process, the window state acquisition unit 16 acquires the window state at the time of the change and the time when the change occurs.

The control unit 22a of the data processing device 20a includes a data acquisition unit 221a, a determination unit 222a, a setting unit 223a, a search unit 224a, and a measurement unit 225a.

The data acquisition unit 221a acquires an input event and a change point on the screen obtained from the screen capture from the terminal device 10 via the communication control unit 21. Further, the data acquisition unit 221a has input events from the terminal device 10a via the communication control unit 21, changes on the screen obtained from the screen capture, and window state data (changes) acquired by the window state acquisition unit 16. The window state at the time of the occurrence and the time when the change occurred) are acquired.

Then, the data acquisition unit 221a stores the input event data of the terminal device 10 and the terminal device 10a in the input event database of the storage unit 23. Further, the data acquisition unit 221a stores the screen capture data including the changed portion on the screen in the screen capture event database of the storage unit 23. Further, the data acquisition unit 221a stores the window state data in the window information database.

The determination unit 222a determines whether or not the window state of the terminal device 10a satisfies the content of the setting information set by the setting unit 223. For example, the determination unit 222a determines whether or not the active window is closed by the terminal device 10a.

The setting unit 223a accepts the input of setting information. An event in the window state to be detected (for example, the active window is closed, etc.) is set in the setting information.

The search unit 224a accesses the input event database and the screen capture database, and searches for the input event of the terminal device 10 and the terminal device 10a corresponding to the window state and the screen capture of the terminal device 10 and the terminal device 10a.

The measurement unit 225a measures the value of the difference between the occurrence time of the input event searched by the search unit 224a and the acquisition time of the screen capture corresponding to the input event. Then, the measurement unit 225a outputs the measured difference as the EtoE processing time.

The storage unit 23a includes an input event database, a screen capture database, and a window state database in a predetermined area.

The input event database is a database that stores the input event data of the terminal device 10 and the terminal device 10a acquired by the data acquisition unit 221a. The screen capture database is a database that stores the screen capture data of the terminal device 10 and the terminal device 10a acquired by the data acquisition unit 221a in the screen capture database. The window state database is a database that stores the window state data acquired from the terminal device 10a by the data acquisition unit 221a.

<Example of processing procedure>
Next, the processes (processes 1, 2, 3, and 4) executed by the system of the third embodiment will be described.

Process 1 is a process in which the system stores input events acquired from each of the terminal device 10 and the terminal device 10a in the input event database. The process 2 is a process in which the system stores the screen capture data acquired from each of the terminal device 10 and the terminal device 10a in the screen capture database. The processes 1 and 2 are processes corresponding to the processes 1 and 2 described with reference to FIG.

Process 3 is a process in which the system stores the window state data acquired by the terminal device 10a in the window state database. The process 4 is a process in which the system calculates the EtoE processing time based on the input event data, the screen capture data, and the window state data obtained in the processes 1, 2, and 3.

Processes 1 and 2 are processes performed in parallel in the background of the terminal device 10. The processes 1, 2, and 3 are processes performed in parallel in the background of the terminal device 10a. Further, the process 4 is started, for example, at the timing instructed by the user of the data processing device 20a. The setting unit 223a of the data processing device 20a shall accept the input of the setting information of the event to be detected in advance.

Since the processes 1 and 2 are almost the same as the above-described embodiments, the description thereof will be omitted and the process 3 will be described.

<Process 3>
In process 3, the system acquires window and process information from the terminal device 10a to which the terminal device 10 is connected at predetermined time intervals, and if there is a change in the window or process, window state data indicating the change is displayed in the window. Store in the status database. For example, if the states of the active windows before and after the terminal device 10a are different and there is no process of the previous active window, the system determines that the window state of the terminal device 10a is "the state in which the active window is closed". Then, the system stores the above window state and the acquisition time of the window state as window state data in the window state database.

Hereinafter, an example of the processing procedure of processing 3 will be described with reference to FIG. 7. For example, the window state acquisition unit 16 of the terminal device 10a acquires information on the active window and the process at predetermined time intervals (S51 in FIG. 7), and if there is a change in the information before and after (Yes in S52), the window state data. To get. Then, when the data acquisition unit 221a of the data processing device 20a acquires the window state data via the communication control unit 21, the acquired window state data is stored in the window state database (S53). After that, if the user of the terminal device 10a does not stop the process and the terminal device 10a is not shut down (No in S54), the process returns to S51. On the other hand, if the user of the terminal device 10a has stopped the process / the terminal device 10 has shut down (Yes in S54), the process is terminated.

<Process 4>
In the process 4, the data processing device 20a accesses the window state database and acquires the window state data (target window state data) of the window state to be detected. Then, the data processing device 20a is based on the acquired target window state data, and the input event data of the input event (target input event on the connection destination terminal side) of the terminal device 10a that occurred immediately before the acquisition time of the window state to be detected. Is obtained from the input event database.

Further, the data processing device 20a screen-captures the screen capture data of the screen capture event (target screen capture event on the connection destination terminal side) of the terminal device 10a that occurs immediately after the acquisition time of the window state shown in the target window state data. Get from the database.

Then, the data processing device 20a acquires the input event data of the target input event on the connection source terminal side corresponding to the target input event on the connection destination terminal side from the input event database. For example, the data processing device 20a corresponds to a target input event on the connection destination terminal side (terminal device 10a side) based on the transition order of the input events of the terminal device 10 and the terminal device 10a in the input event database in time series. , Acquires the input event data of the target input event on the connection source terminal side (terminal device 10 side).

Next, the data processing device 20a sets the time after the occurrence time of the target input event on the terminal device 10 side as the acquisition time from the screen capture data of the terminal device 10 in the screen capture database, and sets the target screen on the terminal device 10a side as the acquisition time. Find the screen capture data that is most similar (or has a similarity greater than or equal to) the screen capture data of the capture event. Then, the data processing device 20a uses the found screen capture data as the screen capture data of the target screen capture event on the terminal device 10 side. Finally, the data processing device 20a outputs the value of the difference between the occurrence time of the target input event on the terminal device 10 side and the acquisition time of the target screen capture event on the terminal device 10 side as the EtoE processing time.

By doing so, the data processing device 20a can calculate the EtoE processing time by using the change in the window state of the terminal device 10 as an event to be detected.

Hereinafter, an example of the process 4 will be described with reference to FIG. For example, the data processing device 20a accesses the window state database, selects the window state data (S61), and determines whether or not the event is a detection target event (S62). For example, the data processing device 20a selects one window state data from the window state database, and determines whether the window state indicated by the selected window state data is the window state to be detected indicated in the setting information. .. Then, when the data processing device 20a determines that the event is a detection target event (Yes in S62), the input event data of the input event on the connection destination terminal side (terminal device 10a side) that occurred immediately before the detection target event is input from the input event database. Acquire (S63). The input event on the connection destination terminal side that occurs immediately before this detection target event is called the target input event on the connection destination terminal side.

After that, the data processing device 20 finds the input event data of the input event on the local terminal side (terminal device 10 side) corresponding to the target input event on the connection destination terminal side (S64). For example, the data processing device 20a corresponds to the target input event on the terminal device 10a side based on the transition order of the input events of the terminal device 10 and the terminal device 10a in the input event database in the time series. Find the input event data for the input event. The data processing device 20a may find the input event data of the input event on the terminal device 10 side having the occurrence time immediately after the occurrence time of the target input event on the terminal device 10a side in the input event database.

Next, the data processing device 20a acquires the screen capture data on the connection destination terminal side generated immediately after the target window state data (immediately after the acquisition time of the window state shown in the target window state data) from the screen capture database. (S65).

After that, the data processing device 20a is a local terminal side (terminal) having a similarity equal to or higher than a threshold value with respect to the screen capture data on the connection destination terminal side acquired in S65 from the screen capture data of the terminal device 10 in the screen capture database. Find the screen capture data of the screen capture of the device 10 side) (S66).

Then, the data processing device 20a determines whether or not the acquisition time of the screen capture in the screen capture data on the local terminal side found in S66 is after the occurrence time of the input event in the input event data on the local terminal side found in S64. Judgment (S67: Is the acquisition time of the screen capture on the local terminal side after the occurrence time of the input event on the local terminal side?).

When the data processing device 20a determines in S67 that the acquisition time of the screen capture in the screen capture data on the local terminal side found in S66 is after the occurrence time of the input event in the input event data on the local terminal side found in S64. (Yes at S67), proceed to S68. Then, the data processing device 20a obtains a value of the difference between the occurrence time of the input event on the local terminal side found in S64 and the acquisition time of the screen capture on the local terminal side found in S66 (S68). Then, the data processing device 20a outputs the value of the difference obtained in S68 (S69). Then proceed to S70.

In S70, when it is determined that the data processing device 20a has confirmed all the window state data in the window state database (Yes in S70), the process ends. On the other hand, when the data processing device 20a determines that any window state data in the window state database is unconfirmed (No in S70), the process returns to S61.

If the data processing device 20a determines in S62 that the window state indicated by the window state data selected in S61 is not a detection target event (No in S62), the process proceeds to S70. Further, in S67, when the data processing device 20a determines that the acquisition time of the screen capture on the local terminal side found in S66 is before the occurrence time of the input event on the local terminal side found in S64 (No in S67). , Proceed to S70.

In each embodiment, the case where the terminal device 10 is remotely connected to the connection destination terminal device (for example, the terminal device 10a) via the VDI server has been described, but the connection destination terminal device is physically connected. It may be a terminal device or a virtual machine provided by VDI. Further, when the terminal device of the connection destination is a terminal device installed in the company, the connection from the terminal device 10 to the terminal device of the connection destination is, for example, an in-house LAN (Local Area) via a VPN (Virtual Private Network) or the like. Network) may be used.

[System configuration, etc.]
Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Further, each processing function performed by each device is realized by a CPU (Central Processing Unit) and a program executed by the CPU, or as hardware by wired logic. Can be realized.

Further, among the processes described in each embodiment, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[program]
The system of each of the above-described embodiments can be implemented by installing a data processing program on a desired computer as packaged software or online software. For example, by causing the information processing apparatus to execute the above data processing program, the information processing apparatus can function as the system of each embodiment. The information processing device referred to here includes a desktop type or notebook type personal computer. In addition, information processing devices include smartphones, mobile communication terminals such as mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDAs (Personal Digital Assistants).

Further, the data processing device of each embodiment can be implemented as a server device in which the terminal device used by the user is a client and the service related to the above processing is provided to the client. In this case, the server device may be implemented as a Web server, or may be implemented as a cloud that provides services related to the above processing by outsourcing.

FIG. 9 is a diagram showing an example of a computer that executes a data processing program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these parts is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, the display 1130.

The hard disk drive 1090 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. That is, the program that defines each process executed by the system of each embodiment is implemented as a program module 1093 in which a code that can be executed by a computer is described. The program module 1093 is stored in, for example, the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration in the system of each embodiment is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD.

Further, the setting information used in the processing of the above-described embodiment is stored as program data 1094 in, for example, a memory 1010 or a hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 into the RAM 1012 and executes them as needed.

The program module 1093 and the program data 1094 are not limited to those stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). Then, the program module 1093 and the program data 1094 may be read from another computer by the CPU 1020 via the network interface 1070.

10,10a Terminal device 11 Input acquisition unit 12 Screen acquisition unit 13 Screen comparison unit 14

Output unit

15,21

Communication control unit

20,20a

Data processing device

22,22a Control unit 23,23a Storage unit 221,221a Data acquisition unit 2222

222a Judgment unit

223,

223a Setting unit

224, 224a Search unit 225, 225a Measurement unit

Claims

Among the events that occur on the connection source terminal of the remote connection, the input unit that accepts the setting information indicating the event to be detected, and
When it is determined that the screen of the connection source terminal has changed due to the event to be detected based on the setting information and the capture of the screen of the connection source terminal acquired at predetermined time intervals, the said case. A data processing device including an output processing unit that outputs a value of a difference between an event occurrence time and a capture acquisition time of a screen in which the change has occurred as an operation delay time.
The output processing unit
The data processing apparatus according to claim 1, wherein the acquisition time of the first screen capture after the occurrence time of the event is used as the acquisition time of the screen capture in which the change occurs due to the event to be detected.
The event to be detected in the setting information is
It is defined by the combination of the operation event of the connection source terminal and the content of the screen change caused by the operation event.
The output processing unit
When the screen change shown in the setting information occurs due to the operation event shown in the setting information, the value of the difference between the occurrence time of the event and the acquisition time of the capture of the screen in which the screen change occurs is set as described above. The data processing apparatus according to claim 1, wherein the data is output as an operation delay time.
The event to be detected in the setting information is
It is an event related to the change of the window state of the connected terminal.
The output processing unit
Changes in the window state of the detection target based on the setting information and the window state of the connection destination terminal acquired at predetermined time intervals in the connection destination terminal which is a terminal remotely connected to the connection source terminal. When it is determined that the screen of the connection source terminal has changed due to
The first process of specifying the operation event in the connection destination terminal that occurred immediately before the time when the change in the window state occurred, and specifying the operation event of the connection source terminal corresponding to the specified operation event. ,
The capture of the screen of the terminal of the connection destination immediately after the time when the change of the window state occurs is specified, and the capture of the screen of the terminal of the connection source having a similarity with the capture of the specified screen is equal to or more than a predetermined value. The second process of specifying and specifying the acquisition time of the capture of the screen of the specified connection source terminal,
When the acquisition time of the screen capture of the connection source terminal specified by the second process is later than the occurrence time of the operation event of the connection source terminal specified by the first process, the operation event The data processing apparatus according to claim 1, wherein the third process of outputting the value of the difference between the occurrence time of the screen and the acquisition time of the capture of the screen as the operation delay time is executed.
A data processing method executed by a data processing device.
Among the events that occur on the connection source terminal of the remote connection, the process of accepting the setting information indicating the event to be detected, and
When it is determined that the screen of the connection source terminal has changed due to the event to be detected based on the setting information and the capture of the screen of the connection source terminal acquired at predetermined time intervals, the said case. A data processing method including a step of outputting a value of a difference between an event occurrence time and a capture acquisition time of a screen in which the change has occurred as an operation delay time.