JP2015170031A - Association method, association program, and association device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To identify drawing processing for an input operation corresponding to a screen operation in a remote desktop system.SOLUTION: Provided is an association method executed by a computer capable of acquiring input operation data and drawing processing data in a remote desktop system in which: an input operation data group and a drawing processing data group in a prescribed period are extracted from the acquired input operation data and drawing processing data; a relative position between a coordinate contained in a piece of input operation data of the input operation data group, and a coordinate in a drawing area contained in a piece of drawing processing data of the drawing processing data group is calculated; on the basis of the relative position of the coordinates, a coordinate assumed to be contained in the input operation data is calculated from a different piece of drawing processing data contained in the drawing processing data group; and on the basis of the assumed coordinate, input operation data contained in the input operation data group is associated with the different piece of drawing processing data.

Description

  The present invention relates to an association method, an association program, and an association apparatus.

  A method for evaluating the performance and quality of an interactive system having a GUI (Graphical User Interface) has been proposed (see, for example, Patent Documents 1 to 3). As an example of the interactive system, there is a remote desktop system in which a GUI of a server is operated from a client device, and a screen generated by a drawing process by the server is provided to the client device. In order to evaluate the performance and quality of a remote desktop system, a response time from when a user performs an operation such as moving a mouse to when a screen is provided may be measured. It can be evaluated that the shorter the response time, the better the quality of the user experience in the remote desktop system.

JP-A-8-153022 JP 2008-191981 JP 2004-38619 A

  In the remote desktop system, if the drawing process for the input operation corresponding to the user's screen operation is one-to-one, the response time is accurately calculated based on the transmission time of the input operation data and the reception time of the drawing process data, and the system Can be evaluated correctly.

  However, the input operation and the drawing process do not always correspond one-on-one. For this reason, even if the response time is calculated simply by associating the input operation data with the drawing processing data, it is difficult to accurately calculate the system quality without accurately calculating the response time.

  Therefore, an object of one aspect is to identify a drawing process for an input operation corresponding to a screen operation in a remote desktop system.

  In one proposal, there is an association method executed by a computer capable of acquiring input operation data including coordinates according to screen operations in a remote desktop system and drawing processing data including coordinates of a drawing processed area. Extracting an input operation data group and a drawing process data group within a predetermined period from the acquired input operation data and drawing process data, and coordinates included in one input operation data of the input operation data group; The relative position of the drawing area coordinate included in one drawing processing data of the drawing processing data group is calculated, and other drawing processing data included in the drawing processing data group based on the relative position of the coordinate The coordinates assumed to be included in the input operation data are calculated from the input operation data included in the input operation data group based on the assumed coordinates. Associating the other drawing processing data, associating method including the process is provided.

  According to one aspect, it is possible to identify a drawing process for an input operation corresponding to a screen operation in the remote desktop system.

The figure which shows the structural example of the remote desktop system which concerns on one Embodiment. The figure which shows the operation example of the remote desktop system which concerns on one Embodiment. The figure which shows an example of the data transmitted / received by a client apparatus and a service server. The figure which shows the example by which a drawing process is performed with respect to input operation. The figure which shows the function structural example of the capture server which concerns on one Embodiment. The figure which shows an example of the information memorize | stored in the operation log table which concerns on one Embodiment. The figure which shows an example of the information memorize | stored in the user operation table which concerns on one Embodiment. The figure which shows an example of the information memorize | stored in the drawing process table which concerns on one Embodiment. The figure which shows an example of the relative coordinate which concerns on one Embodiment. The figure which shows an example of the coordinate of the assumed cursor which concerns on one Embodiment. The figure which shows an example of the matching which concerns on one Embodiment. The flowchart which shows the matching process concerning 1st Embodiment. The figure which shows an example of the information memorize | stored in the operation log table concerning 1st Embodiment. The figure which shows an example of the relative coordinate concerning 1st Embodiment, and the coordinate of the assumed cursor. The figure for demonstrating the matching process concerning 2nd Embodiment. The figure for demonstrating the matching process concerning 2nd Embodiment. The figure which shows an example of the information memorize | stored in the operation log table concerning 2nd Embodiment. The flowchart which shows the matching process concerning 2nd Embodiment. The figure which shows an example of the information memorize | stored in the response time table which concerns on one Embodiment. The figure which shows an example of the hardware constitutions of the capture server which concerns on one Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, the duplicate description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[Remote desktop system]
First, a remote desktop system according to an embodiment of the present invention will be briefly described with reference to FIGS. 1 and 2. FIG. 1 shows a configuration example of a remote desktop system according to an embodiment. FIG. 2 shows an operation example of the remote desktop system according to the embodiment.

  The remote desktop system 10 of this embodiment includes a client device 1, a service server 2, a switch 3, and a capture server 4. The client device 1 and the service server 2 are connected to each other via a network 5. The capture server 4 is a device that can acquire data transmitted and received between the client device 1 and the service server 2 via the switch 3.

  For example, the remote desktop system 10 includes a system called a thin client system that the service server 2 generates a desktop screen and provides to the client device 1.

  In the system of the present embodiment, there are a plurality of client devices 1, but a single client device 1 may be used. In the system of this embodiment, the service server 2 and the capture server 4 are singular, but may be plural. When there are a plurality of service servers 2 and capture servers 4, it is possible to perform distributed processing.

  The client device 1 and the service server 2 repeat the following series of processes as shown in FIG. First, the client device 1 transmits an input operation command to the service server 2 in response to an operation of an input device such as a mouse (hereinafter also referred to as “user operation”) by the user (step S1). The input operation command includes input operation data indicating information related to the operation.

  The “user operation” is an operation by the user such as a mouse button depression, a mouse button movement, or a key input. The client device 1 sends an input operation command corresponding to the user operation to the service server independently of the screen update. 2 to send.

  The service server 2 performs processing corresponding to the received input operation command. The service server 2 returns a drawing process command to the client device 1 (step S2). The drawing processing command includes screen drawing processing data generated by the drawing processing by the service server 2. Note that as an example of the drawing process by the service server 2, various drawing processes executed by the computer in response to the input operation, such as display of characters in response to key input, movement of a pointer accompanying a mouse movement operation, scrolling of the screen, and the like. Is mentioned.

  The client device 1 updates the display screen based on the drawing processing data received from the service server 2. Thereby, the user can visually recognize the result of the user operation from the display on the screen.

  In the remote desktop environment, the shorter the time from when the “input operation data” is transmitted from the client device 1 until the client device 1 receives the “drawing process data”, the better the quality experienced by the user. The system includes a switch 3 and a capture server 4 for acquiring input operation data and drawing processing data transmitted and received between the client device 1 and the service server 2 in time series.

  The switch 3 has a normal switching function and a port mirroring function. The switch 3 port mirrors the input operation data transmitted from the client device 1 to the service server 2 and the drawing processing data returned from the service server 2 to the client device 1 with respect to the capture server 4.

  The capture server 4 acquires the input operation data and the drawing process data that are port-mirrored by the switch 3. The capture server 4 uses the captured input operation data and drawing processing data to transmit the input operation in the client device 1, and then returns the drawing processing data generated according to the input operation to the client device 1. The response time to be calculated is calculated.

(Example of data sent and received between client device and service server)
An example of data transmitted and received between the client device 1 and the service server 2 will be specifically described with reference to FIG. In the example illustrated in FIG. 3, the arrow pointing from the bottom to the top indicates data transmitted from the client device 1 to the service server 2. Data transmitted from the client device 1 to the service server 2 includes input operation data.

  On the other hand, arrows pointing from top to bottom indicate data transmitted from the service server 2 to the client device 1. Data transmitted from the service server 2 to the client device 1 includes drawing processing data. The drawing processing data includes screen data to be drawn when the drawing processing involves drawing a new screen. Here, the frame transmission completion command indicates that transmission of the drawing processing data for displaying one frame of the screen has been completed. In other words, the drawing processing data transmitted between one frame transmission completion command and the next frame transmission completion command corresponds to drawing processing data for displaying a screen for one frame.

  Incidentally, in principle, the service server 2 does not transmit the frame transmission completion command data when the screen displayed on the client device 1 is not updated. For example, as illustrated in FIG. 3, when the user performs an operation of moving the window with the mouse, the “drawing processing data” corresponding to the input operation data of the client device 1 in response to the user operation “pushing and pushing the mouse button” "Corresponds one to one. On the other hand, in the communication data indicated by A, “drawing process data” is only transmitted once for a plurality of input operation data corresponding to a change in a plurality of coordinates indicating a user operation “movement of the mouse cursor”.

  That is, in the cursor position information by the mouse operation, the input operation data for the user operation is often notified from the client device 1 to the service server 2 in detail. As a result, the capture server 4 acquires more input operation data based on a user operation than drawing processing data in an operation of moving a window with a mouse.

  Thus, in a remote desktop environment, a plurality of input operations may be thinned out and combined into a single drawing process. FIG. 4A shows an example in which drawing processing is performed by thinning out a plurality of input operations. For example, the user operations a to d are not drawn by performing the drawing process for each input operation corresponding to the user operation, but are collectively executed as a single drawing process. It is drawn at the time. Similarly, the user operations e to i, the user operations j to n, and the user operations o to s are collectively performed at a time point (2), (3), and (4), respectively. Drawn.

  In addition, in the remote desktop environment, drawing processing for user operations is somewhat delayed. Therefore, it is necessary to provide a margin for the user operation in the end section of the drawing process to be extracted. Generally, the limit value of the waiting time of response time for interactive input operation is said to be 150 milliseconds, which is considered to be usable as the shortest value of time α. For the purpose of abnormality detection, a value in units of several seconds is considered appropriate.

  In addition, when the delay resulting from a network has generate | occur | produced, it is necessary to further extend a termination | terminus area according to the generated delay time separately from the reference | standard of said waiting time. FIG. 4B shows a time-series relationship between the user operation and the drawing process. As described in the present embodiment, in the remote desktop environment, input operation data corresponding to a user operation is sent to the service server 2 through the network 5. After the drawing process, the service server 2 transmits drawing processing data as a processing result to the client device 1 through the network 5. Therefore, the response time always includes a delay time due to data communication. In the example of FIG. 4B, the drawing process data for the user operation a is acquired at the timing indicated by (1) due to the delay. The following user operation b is performed at the timing of (1). Hereinafter, the user operation b is performed at the timing indicated by (2), and the user operation c is performed at the timing indicated by (3).

  As described above, in the remote desktop environment, the input operation data and the drawing processing data are not necessarily transmitted and received in a one-to-one correspondence. Further, it is not of a nature that specific drawing processing data is always returned for the same input operation data. For this reason, even if the response time is calculated simply by associating the input operation data with the drawing processing data, it is difficult to accurately calculate the system quality without accurately calculating the response time.

  Therefore, the capture server 4 can identify drawing processing data corresponding to one input operation data from the drawing processing data group received by the client device 1 from the service server 2.

  Hereinafter, an embodiment of the association between the input operation data executed by the capture server 4 and the drawing processing data will be described. The capture server 4 is an example of an association device that associates input operation data with drawing processing data. Another example of the associating device is another server that can obtain data from the capture server 4. That is, the associating device may be incorporated in the system in any form as long as it can acquire data transmitted and received between the client device 1 and the service server 2.

  Also, in this specification, “window movement” by “mouse movement” will be described as an example of a user operation. However, in the association method according to the present embodiment, the user operation is not limited to moving the window.

[Capture server function configuration]
Next, an example of a functional configuration of the capture server 4 according to an embodiment will be described with reference to FIG. FIG. 5 shows an example of the functional configuration of the capture server 4 according to an embodiment. The capture server 4 includes a capture unit 11, an L7 analysis unit 12, a data extraction unit 13, an assumed position calculation unit 14, an assumed position calculation unit 15, an association unit 16, a response time calculation unit 17, and a storage unit 20. The storage unit 20 of the capture server 4 stores an operation log table 21, a user operation table 22, a drawing process table 23, and a response time table 24.

  The capture unit 11 acquires input operation data transmitted from the client device 1 and drawing processing data transmitted from the service server 2, which are port mirrored by the switch 3. At this point, the input operation data and the drawing process data are binary data.

  The L7 analysis unit 12 analyzes the input operation data and the drawing processing data acquired by the capture unit 11, converts the data, generates an operation log indicating the contents of the input operation data and the drawing processing data, and an operation log table Write to 21.

  Note that the capture unit 11 and the L7 analysis unit 12 correspond to a data acquisition unit that acquires and analyzes input operation data and drawing processing data. The input operation data includes coordinates on the screen according to the user operation. The drawing process data includes the coordinates of the drawing area that has been drawn by the service server 2 in accordance with the input operation data.

  The data extraction unit 13 extracts an input operation data group and a drawing process data group within a predetermined period from the data written in the operation log table 21. As an example of the predetermined period, there is a time obtained by adding a time α to the time from when the mouse button is pressed down by the user operation, the cursor moves, and the mouse button is pushed up (released). As an example of the time α, a delay until data acquisition between the client device 1 and the service server 2 such as a time required for drawing processing in the service server 2 and a time required for transferring input operation data and drawing processing data via the network 5. Time is given. The extracted input operation data group is stored in the user operation table 22, and the extracted drawing process data group is stored in the drawing process table 23.

  The assumed position calculation unit 14 extracts one input operation data from the extracted input operation data group. The assumed position calculation unit 14 extracts drawing process data corresponding to the extracted input operation data from the extracted drawing process data group. The assumed position calculation unit 14 calculates the relative position of the coordinates of the drawing area with respect to the coordinates of the mouse cursor from the coordinates of the mouse cursor included in the extracted input operation data and the coordinates of the drawing area of the drawing processing data.

  As will be described later, the assumed position calculation unit 14 may extract the last input operation data of the input operation data group and the last drawing process data of the drawing process data group, and calculate the relative position from the extracted data. . Further, the assumed position calculation unit 14 takes out the input operation data one or more previous times from the last of the input operation data group and the drawing process data one or more times before or from the last of the drawing process data group, and takes them out. The relative position may be calculated from the data.

  For example, a drawing process corresponding to an input operation such as “pressing a mouse button” has a one-to-one correspondence. The reason is that, in the case of scroll drawing by pressing down a mouse button, the speed at which a human user can operate the keyboard is sufficiently slower than the processing capability of the computer. On the other hand, the position information of the cursor by the mouse operation is often notified in detail of the input operation. In that case, the drawing process for the input operation may correspond many-to-one. Therefore, the assumed position calculation unit 14 obtains input operation data and drawing processing data corresponding to a drawing process corresponding to an input operation such as “mouse button depression” from the input operation data group and the drawing processing data group. It is preferable to take out and calculate the relative position.

  The assumed position calculation unit 15 calculates coordinates assumed to be included in the input operation data from other drawing process data included in the drawing process data group based on the calculated relative position of the coordinates.

  The association unit 16 associates the input operation data included in the input operation data group with the other drawing processing data based on the assumed coordinates.

  The response time calculation unit 17 calculates a response time that is a time difference between the transmission time of the input operation data associated with the association unit 16 and the transmission time of the drawing process data.

  Next, the data structure of various data stored in the storage unit 20 of the capture server 4 will be described.

  The operation log table 21 is a table in which operation log information of data transmitted and received between the client device 1 and the service server 2 is recorded. An example of operation log information stored in the operation log table 21 is shown in FIG. The operation log table 21 includes items of time information (Time) 21a, client information 21b, server information 21c, type (Type) 21d, command (Command Name) 21e, and command argument (Args) 21f. The time information 21a indicates the time when data is transmitted. The client information 21b indicates the IP address (Client Address) of the client device 1. The client information 21b may include a port number (Client Port) of the client device 1. The server information 21c indicates the IP address (Server Address) of the service server 2. The server information 21c may include a port number (Server Port) of the service server 2. The type information 21d indicates whether the transmitted data is input operation data (REQ) or drawing processing data (RES). The command 21e indicates the instruction content of the input operation data or the drawing process data, and the command argument 21f indicates the argument of the command 21e.

  Note that, in the operation log information shown in FIG. 6A, the region copy drawing process (T) and the image from the service server 2 in response to an input operation (S) for pressing the mouse button in the scroll bar area. FIG. 6B shows an example in which the drawing process (U) for drawing is transmitted. The left screen in FIG. 6B shows a state immediately before the mouse button is pressed, and the right screen in FIG. 6B shows a state immediately after the mouse button is pressed.

  When the mouse button is pressed (S), the screen scrolls upward as shown in the left screen of FIG. 6B. In this way, the change in the screen that the screen scrolls upward is specified by copying a portion of the screen from the bottom to the top (T), and the screen is newly displayed in the region below the copied portion. It can be expressed by drawing (U). “Upward area copy (T)” and “new drawing (U)” are both drawing processes that occur when the screen scrolls upward.

  In the remote desktop system 10, drawing processing data for screen update is sent from the service server 2 to the client device 1, and the client device 1 updates the screen display according to the received data. At that time, as shown in FIG. 6B, the screen update is not generally performed on the entire screen, but only a portion that needs to be updated is divided into small sections, and the group of small sections is transmitted to the client device 1. The client device 1 configures the updated screen by reflecting the small section data on the screen display according to the coordinates. At this time, a command for notifying how far data is to be displayed on the screen at that time is determined. As an example, the above-mentioned “frame transmission complete” command is used, and a screen display constituted by a data group sandwiched between them (and a screen displayed up to that point) is called a frame.

  Specifically, the data drawn in the immediately preceding frame is not retransmitted, but the copy source coordinates and size and the copy destination coordinates for the immediately preceding frame are transmitted to the client device 1. The client device 1 collects image data of a specified size from the specified copy source coordinates and size, and displays it at the specified copy destination coordinates. Thus, the screen can be updated without sending all the image data of the screen. In the present embodiment, this is executed by the “area copy” command. Examples of usage include scroll processing.

  The user operation table 22 records a list of extracted input operation data groups within a predetermined period. FIG. 7 shows an example of information stored in the user operation table according to an embodiment. In FIG. 7, “window movement with mouse (mouse movement)” is given as an example of user operation. In this case, the input operation data stored in the user operation table 22 includes items of transmission time information 22a of the input operation data and coordinates (X, Y) 22b on the screen corresponding to “move window by mouse”.

  When the window is moved by the mouse, commands for user operations are generated in the order of “mouse button press” → “mouse cursor move (plural)” → “mouse button press”.

  Therefore, the user operation table 22 according to the present embodiment extracts an input operation data group of a “mouse cursor movement” command generated within a predetermined period from the operation log information accumulated in the operation log table 21, and the input operation data Coordinates included in the data are listed in time series (user operation list).

  The drawing process table 23 extracts drawing process data groups of the “area copy” command generated within a predetermined period from the operation log information accumulated in the operation log table 21, and lists them in time series (draw process list). ). FIG. 8 shows an example of drawing process data stored in the drawing process table 23 according to the embodiment. For example, the drawing processing data includes items of transmission time information 23a of drawing processing data and coordinates 23b of the copy source (X, Y) and coordinates 23c of the copy destination (X, Y).

Here, the predetermined period is a time period from when an input operation corresponding to a user's mouse button pressing operation occurs until the mouse button is pressed up (mouse button is released) (for example,
It may be 150 milliseconds, which is a limit value of the waiting time of response time for interactive input operation, or may be a time obtained by adding time α to this time. The time α may be determined in consideration of a delay time for drawing processing and data communication.

  In the response time table 24, a response time that is a time difference between the transmission time of the input operation data and the reception time of the drawing process data is recorded.

[Association between input operation data and drawing processing data]
As shown in FIG. 3, when the user operation is such as scroll drawing by “pressing and pushing the mouse button (cursor key)”, input operation data and drawing processing data of the client device 1 according to the user operation, Corresponds one-to-one. This is because the speed at which a human user can operate the mouse and keyboard is sufficiently slower than the processing capability of the computer.

  Therefore, it is preferable that the end of the predetermined period when the data extraction unit 13 extracts the input operation data group and the drawing process data group is not the completion of the “mouse movement” command but the “mouse button press” command or later. . According to this, since the drawing process corresponding to the last input operation of the input operation data group surely occurs, the section in which the last input operation data and the last drawing process data of the drawing process data group correspond one-to-one. Can be extracted.

  In the method for associating input operation data and drawing processing data according to the first and second embodiments described below, user operation (input operation) and drawing processing are associated with each other in the following three steps.

(Step 1: Extract user operation range)
Step 1 extracts a range of user operations. In step 1, relative coordinates are calculated from the coordinates of the mouse cursor and the coordinates of the window to be drawn. An example of the calculation of relative coordinates will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of relative coordinates according to an embodiment.

  In step 1, drawing is performed from the coordinates of the “mouse move” command of the last input operation data of the input operation data group and the coordinates of the drawing area of the “copy area” command of the last drawing processing data of the drawing process data group. The relative coordinates between the window and the mouse cursor are calculated. In the upper diagram of FIG. 9, among the last coordinates (x, y) of the “move mouse” command and the coordinates (x1, y1, x2, y2, x3, y3) of the last area in the coordinates of the “area copy” The relative coordinates of the copy destination window and the mouse cursor are calculated from the upper left coordinates (x2, y2) of the copy destination.

  The coordinates (x1, y1) are the upper left coordinates of the copy source window, the coordinates (x2, y2) are the upper left coordinates of the copy destination window, and the coordinates (x3, y3) are the copy destination. The coordinates of the lower right corner of the window. Of the coordinates of the “copy area” command in FIG. 8, the copy destination coordinates (X, Y) 23c correspond to the coordinates (x2, y2) at the upper left of the window. In the example of the coordinates of the “copy area” command in FIG. 8, the coordinates (x3, y3) at the lower right of the window are not shown.

  For example, an example of the input operation data group shown in FIG. 7 and an example of the drawing process data group shown in FIG. 8 will be described. The last coordinates (x, y) of the coordinates 22b of the “move mouse” command as an example of the input operation shown in FIG. 7 are (556, 410). Further, the upper left coordinates (x2, y2) of the last area of the “area copy” coordinates 23b and 23c, which is the drawing processing example shown in FIG. 8, that is, the copy destination coordinates (X, Y) 23c is (229). 401). Therefore, as shown in the lower diagram of FIG. 9, the drawing is performed from the last coordinates (556, 410) of “Mouse move” and the upper left coordinates (229, 401) of the last area in the coordinates of “Area copy”. The relative coordinates between the window to be displayed and the mouse cursor are calculated as (327, 9).

(Step 2: Extract the drawing process range)
Step 2 extracts a drawing process range. In step 2, the coordinates of the mouse cursor assumed for each of the drawing areas (logical coordinates where the corresponding mouse cursor should be) are calculated based on the relative coordinates calculated in step 1. An example of calculating the coordinates of the assumed mouse cursor from the drawing processing data group will be described with reference to FIG. FIG. 10 shows an example of the coordinates of the assumed mouse cursor according to the embodiment.

  In step 2, the assumed coordinates of the mouse cursor are calculated by adding the relative coordinates to the coordinates 23c of the “copy area” command. For example, the assumption shown in the right table of FIG. 10 is made by adding the relative coordinates (327, 9) to the coordinates (432, 406) which are the first coordinates 23c of the “copy region” command in the left table of FIG. The coordinates (759, 415) of the mouse cursor to be executed are calculated. In this manner, the coordinates of the assumed mouse cursor can be calculated from the coordinates 23c of all “area copy” commands in the acquired drawing processing data group.

  From the above, in step 2, the thinning shown in FIG. 4A is considered, that is, the influence of the thinning is removed in consideration of the fact that the input operation data and the drawing processing data do not correspond one-to-one. Let us assume the coordinates of the mouse cursor from the coordinates of the drawing processing data.

(Step 3: Associating user processing with drawing processing)
Step 3 associates user processing with drawing processing. In step 3, the input operation data (user operation) and the drawing process data are associated with each other from the assumed mouse cursor coordinates calculated in step 2 and the mouse cursor coordinates included in the input operation data. . FIG. 11 shows an example of the result of associating the assumed mouse cursor position calculated in step 2 with the coordinates of the “mouse move” command in the input operation data group of FIG. Here, a “move mouse” command having coordinates that match the assumed mouse cursor position is searched from the user operation table 22, and the matched “mouse move” command is associated with the corresponding drawing process data. The right table of FIG. 11 shows the associated “mouse move” command filled in. With this association, the drawing process for the input operation can be identified in the remote desktop system 10.

  In the association method according to the present embodiment, the response time in the remote desktop system 10 can be accurately measured by the above three steps. Thereby, the performance and quality of the remote desktop system 10 can be correctly evaluated based on the response time. Hereinafter, the associating method according to the present embodiment will be described in detail for the first embodiment and the second embodiment.

<First Embodiment>
[Association process executed on the capture server]
Next, an example of the association process executed by the capture server 4 according to the first embodiment will be described in detail with reference to FIGS. FIG. 12 is a flowchart showing the association processing according to the first embodiment. FIG. 13 shows an example of information stored in the operation log table according to the first embodiment. FIG. 14 shows an example of relative coordinates and assumed cursor coordinates according to the first embodiment. Here, a case where the user performs an operation of moving a window on the screen with a mouse cursor will be described as an example. That is, the association processing executed by the capture server 4 in response to an operation in which the mouse button is pressed down by a user operation, the mouse cursor moves, and the mouse button is pressed up (released) will be described.

  When the association process of FIG. 12 is started, the capture unit 11 acquires the input operation data transmitted from the client device 1 and the drawing process data transmitted from the service server 2. The L7 analysis unit 12 analyzes the binary data acquired by the capture unit 11, converts the data, and generates input operation data information and drawing processing data information. The storage unit 20 stores the generated input operation data and drawing process data in the operation log table 21 (step S10). An example of the operation log table 21 is shown in FIG. In the operation log table 21, the command “mouse move” instructs “mouse movement” by the user. The “command argument” of the “move mouse” command is a coordinate indicating the position of the cursor included in the input operation data of the “move mouse” command.

  The command “copy rect cmd” instructs to copy the area of the window to be drawn included in the drawing process data of the “area copy” command. In the “command argument” of the “area copy” command, the coordinates of “upper left of copy source”, “upper left of copy destination”, and “lower right of copy destination” of the window to be moved are specified in order.

The command “send desk picture” instructs to copy the area of the window to be drawn included in the drawing process data of the “new drawing” command. "Image format", "Upper left coordinates of drawing area", "Image data width and height" are specified in order for "Command argument" of "New drawing" command. The command "send desk flush image" is actually Is instructed to start display drawing on the client device 1.

  The command “mouse button down / mouse button up” indicates an operation of “pressing / pressing down a mouse button” by the user. The “command argument” of the “press / press mouse button” command notifies which button is pressed and pressed. “Btn = 0” represents the left button.

  The data extraction unit 13 extracts one input operation data from the operation log table 21 (step S11). Thereby, any of the input operation data shown in each row of FIG. 13 is extracted. Whether it is input operation data can be determined by the type included in each input operation data of the operation log table 21 of FIG. The data extraction unit 13 sequentially extracts the input operation data (step S11) until it is determined that the extracted input operation data is a “mouse button depression” operation (step S12). If the extracted input operation data is a “mouse button depression” operation, the process proceeds to step S13. Then, in step S13, the storage unit 20 extracts input operation data from the operation log table 21 and creates a user operation list until the input operation data that has been extracted is a “mouse button push-up” operation (step S13). S14). Next, the data extraction unit 13 determines whether all the data stored in the operation log table 21 has been processed (step S15). If all the data has not been processed, the process returns to step S11 for the next input. The operation data is taken out and the processes of steps S12 to S15 are executed.

  Thus, the “mouse move” command group (““ button push command on the second line from the top ”) and“ button push command on the third line from the bottom ”in the user operation table 22 shown in FIG. The “move mouse” command group) is extracted. Thus, by extracting the “move mouse” command group, the user who has extracted the range of input operation (user operation) data corresponding to step 1 out of the three steps of the association method of the present embodiment described above. An operation list is created. The user operation list corresponds to an input operation data group.

  When all the data has been processed, the data extraction unit 13 extracts one input operation data from the user operation list stored in the user operation table 22 (step S16).

  Next, the data extraction unit 13 acquires the start time from the time information of the extracted input operation data group (step S17). Next, the data extraction unit 13 acquires the final time of the drawing process to be acquired from the time information of the final input operation data of the extracted input operation data group (step S18).

  Next, the data extraction unit 13 creates a drawing process list in which the “copy rect cmd” command group (“area copy” command group) existing between the acquired start time and end time is extracted (step S19). . By extracting the “copy area” command group in this way, a drawing processing list is created by extracting the range of drawing processing data corresponding to step 2 out of the three steps of the association method of the present embodiment described above. The In the example of FIG. 13, a drawing processing list is created by extracting the range of image processing data of the “copy rect cmd” command existing within a certain range from the “button push up” command on the third line from the bottom. The drawing process list corresponds to a drawing process data group.

  Next, the assumed position calculation unit 14 acquires coordinates included in the final image processing data in the drawing processing list (step S20). Next, the assumed position calculation unit 14 acquires coordinates included in the final input operation data in the user operation list (step S21). Next, the assumed position calculation unit 14 calculates a relative seating amount from the upper left coordinates included in the final image processing data of the drawing processing list and the coordinates included in the final input operation data of the user operation list (Step S1). S22).

  According to the example of FIG. 13, when the “move mouse” command up to the “button up” command in (1) is viewed, the coordinates (X, Y) of the final input operation data (2) are (506, 18). When the extracted “copy area” command is viewed, the coordinates (X2, Y2) at the upper left of the window of the final image processing data (3) are (10, 8). Thereby, as shown in FIG. 14A, the relative position is calculated as (496, 10).

  Next, the assumed position calculation unit 15 calculates a logical mouse cursor position (assumed mouse cursor position) for the drawing process data listed in the drawing process list (step S23). For example, in the example of FIG. 13, among the drawing process data other than the drawing process data (3) of the last “area copy” command, the drawing process data (4), (6), For (7), the assumed mouse cursor position is calculated. For example, in the case of the drawing processing data (4), the upper left coordinates (X, Y) of the copy destination of the window area are (365, 251). As shown in FIG. 14B, by adding the previously obtained relative coordinates (496, 10) to the coordinates (365, 251), coordinates assumed as the logical position of the mouse cursor. (861,261) is calculated.

  Returning to FIG. 12, the associating unit 16 extracts one drawing process data from the drawing process list (step S24). The associating unit 16 searches the user operation table 22 for the input operation data of the “move mouse” command that matches the assumed coordinates. The associating unit 16 associates the input operation data with the drawing processing data on the assumption that the drawing operation is executed by the input operation if the input operation data matching the assumed coordinates is stored in the user operation table 22 ( Step S25). In the example of FIG. 13, the associating unit 16 sets the input operation data to be associating with the “copy rect cmd (region copy)” command of the image processing data (4) in time series. Only the “mouse move” command is required. In the example of FIG. 13, the input operation data (5) of the “move mouse” command that matches the coordinates assumed from the drawing processing data (4) is associated.

  Next, the associating unit 16 deletes entries (items) existing before in time series from the associated operations from the extracted operation group (step S26). In the example of FIG. 13, the input operation data of the “mouse move” command existing in chronological order before the image processing data (4) is deleted. Although step S26 can be omitted, it is better not to omit it in order to use the operation log table 21 effectively.

  Thereafter, in step S27, the assumed position calculation unit 15 calculates the assumed coordinates of the mouse cursor that are assumed for the drawing process data for which the assumed coordinates are not included, which are included in the drawing process list. Then, the association unit 16 performs the association process of steps S24 to S27 until the process is performed for all the drawing process data included in the drawing process list (step S28). Accordingly, in the example of FIG. 13, the coordinates assumed as the logical position of the mouse cursor are similarly calculated for the drawing processing data (6) and (7), and the “mouse” matching the assumed coordinates is calculated. Is associated with the input operation data of the "Move" command.

  For all the drawing process data included in the drawing process list (step S28), until the process of associating with any of the input operation data included in the user operation list is completed (step S29), the association process of S24 to S28. Is executed. When all the association processes are completed, this process ends.

  As described above, according to the capture server 4 according to the first embodiment, the drawing process from the service server 2 for the input operation by the client device 1 can be identified in the remote desktop system 10. Thereby, the input operation data can be associated with the drawing process data obtained as a result of the drawing process corresponding to the input operation. That is, the capture server 4 calculates a relative coordinate that is a difference between the coordinate information of the “input operation according to the user operation” and the “drawing process”, calculates the coordinate of the assumed mouse cursor using the relative coordinate, The input operation data having coordinates that match the calculated coordinates of the mouse cursor are specified. The input operation data specified in this way is associated with the drawing process data. The response time can be accurately measured from the difference in time information between the input operation data and the drawing process data thus associated. As a result, it is possible to correctly evaluate the performance and quality of the remote desktop system 10 based on the accurate response time.

Second Embodiment
Next, an example of the association process executed by the capture server 4 according to the second embodiment will be described in detail with reference to FIGS. 15 and 16 are diagrams for explaining the association processing according to the second embodiment. FIG. 17 shows an example of information stored in the operation log table according to the second embodiment. FIG. 18 is a flowchart illustrating the association processing according to the second embodiment. Here, as in the first embodiment, a case where the user performs an operation of moving a window on the screen using a mouse cursor will be described as an example.

  In the first embodiment, the relative coordinates are obtained with reference to the upper left coordinates of the copy destination area included in the drawing process data of the “area copy” command. However, as shown in FIG. 15, when the left edge of the window exceeds the left edge of the screen when the mouse is moved, a portion outside the screen range is not notified as the copy destination area. For this reason, the relative position of the mouse cursor from the coordinate Q with respect to the upper left of the screen in the window area is incorrect even though the relative position of the mouse cursor from the coordinate P at the upper left of the window is the correct coordinate of the mouse cursor. May be calculated.

  Therefore, in the second embodiment, the assumed position calculation unit 14 calculates the relative position of the mouse cursor from the upper left coordinates of the window, as in the association method according to the first embodiment. In addition, in the second embodiment, the assumed position calculation unit 14 calculates the relative position of the mouse cursor from the lower right coordinates of the window.

  For example, according to the example of FIG. 13, when the “move mouse” command up to the “button up” command in (1) is viewed, the coordinates (X, Y) of the final input operation data (2) are ( 506, 18). In the drawing process data (3) of the extracted “copy rect cmd” command, the coordinates (x2, y2) at the upper left of the window are (10, 8). Therefore, as shown in the right coordinate of FIG. 15, the relative position of the mouse cursor from the upper left coordinate of the window is calculated as (496, 10) as described in the first embodiment.

  In addition, in the second embodiment, the relative position of the mouse cursor is calculated from the lower right coordinates of the window. In the drawing processing data (3) of the extracted “copy rect cmd” command, the coordinates (x3, y3) at the lower right of the window are (942, 593). Therefore, the relative position of the mouse cursor with respect to the lower right coordinates of the window is calculated as (436, 575).

  Here, as shown in FIG. 16, it is assumed that the left 300 dots of the window are out of the screen. At this time, the x coordinate of the left end of the copy destination included in the drawing process data of the “area copy” command is set to zero. That is, the upper left coordinates (x2, y2) of the copy destination of the “copy area” command are set to (0, 8). For example, in the second embodiment, it is assumed that input operation data and drawing process data stored in the operation log table 21 illustrated in FIG. 17 are used. In this case, the difference between the input operation data and the drawing process data stored in the operation log table 21 in the first embodiment is only the difference data described in the central portion. The “copy area” command in the difference data notifies that the lower right coordinate coordinates (x3, y3) of the copy destination are (632, 593).

  When the coordinates of the mouse cursor that are assumed are calculated based on the upper left coordinates included in the above-mentioned “copy area” command, (496, 18) is obtained. The actual correct coordinates are (196, 18), and the drawing process data and the input operation data are not correctly associated.

  In this case, when the assumed coordinates of the mouse cursor are calculated based on the lower right coordinates included in the “copy area” command, (206, 18), the correct mouse cursor coordinates are obtained. Therefore, in the second embodiment, the coordinates of the assumed mouse cursor are calculated using a plurality of coordinates that define the window area, and the input operation data and the drawing process data are more accurately associated with each other.

[Association process executed on the capture server]
As shown in FIG. 18, the association process according to the second embodiment is different from the association process according to the first embodiment only in steps S100 and S101. Therefore, the same steps as those shown in the flowchart of the association process according to the first embodiment in FIG. 12 are given to the same parts as the association process according to the first embodiment. That is, steps S10 to S22, steps S23 to S25, and steps S26 to S29 shown in FIG. 18 are the same processes as steps S10 to S29 in FIG.

  In step S100, the assumed position calculation unit 14 calculates relative coordinates (lower right reference) from the lower right coordinates included in the final image processing data of the drawing processing list and the coordinates included in the final input operation data of the user operation list. Relative coordinates). Next, steps S23 to S25 are executed, and the associating unit 16 associates the input operation data that matches the coordinates of the assumed mouse cursor with the upper left reference calculated based on the upper left reference relative coordinates in step S22. However, there may be cases where there is no input operation data that matches the assumed coordinates of the mouse cursor based on the upper left. Even in that case, in the second embodiment, in step S101, the associating unit 16 matches the coordinates of the assumed mouse cursor with the lower right reference calculated based on the relative coordinates with the lower right reference. Associate operation data.

  As described above, according to the capture server 4 according to the second embodiment, “input operation based on user operation” acquired by the capture server 4 based on a plurality of logical coordinates (assumed coordinates). The “drawing process” can be associated more accurately. The response time can be measured more accurately from the difference in time information between the “input operation data” and the “drawing process data” thus associated. As a result, the accuracy of the performance and quality evaluation of the remote desktop system 10 can be further increased based on the response time.

  FIG. 19 shows an example in which the response time is calculated from the difference in time information between the input operation data and the drawing process data associated with each other in the above embodiments. FIG. 19 shows an example of information stored in the response time table. The response time calculation unit 17 calculates the response time from the input operation of “movement of the mouse” to the actual screen drawing from the difference in time information between the input operation data and the drawing processing data. For example, the performance and quality of the remote desktop system 10 may be evaluated based on a representative value such as an average value or a median value of these response times.

(Hardware configuration example)
Finally, a hardware configuration example of the capture server 4 according to an embodiment will be briefly described. FIG. 20 shows an example of the hardware configuration of the capture server 4 according to an embodiment.

  The capture server 4 includes an input device 101, a display device 102, an external I / F 103, a RAM (Random Access Memory) 104, a ROM (Read Only Memory) 105, a CPU (Central Processing Unit) 106, a communication I / F 107, and an HDD (Hard). Disk Drive) 108 are connected to each other by a bus B.

  The input device 101 includes a keyboard, a mouse, and the like, and is used to input each operation to the capture server 4 according to a user's mouse operation or the like. The display device 102 includes a display or the like, and displays a system operation result or the like to the administrator of the capture server 4.

  The communication I / F 107 is an interface that connects the capture server 4 to the network 5. Thereby, the capture server 4 can acquire input operation data and drawing process data via the communication I / F 107.

  The HDD 108 is a non-volatile storage device that stores programs and data. The stored programs and data include an OS (Operating System) that is basic software for controlling the entire apparatus, and application software that provides various functions on the OS. Further, the HDD 108 stores an association program executed by the CPU 106 in order to perform the association process of the above embodiment.

  The external I / F 103 is an interface with an external device. The external device includes a recording medium 103a. The capture server 4 can read and / or write to the recording medium 103 a via the external I / F 103. The recording medium 103a includes a CD (Compact Disk), a DVD (Digital Versatile Disk), an SD memory card (SD Memory card), a USB memory (Universal Serial Bus memory), and the like.

  The ROM 105 is a nonvolatile semiconductor memory (storage device), and stores programs and data such as BIOS (Basic Input / Output System), OS settings, and network settings that are executed at startup. The RAM 104 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The CPU 106 is an arithmetic device that realizes control and mounting functions of the entire apparatus by reading programs and data from the storage device (for example, “HDD” and “ROM”) onto the RAM and executing processing.

  Each of the data extraction unit 13, the assumed position calculation unit 14, the assumed position calculation unit 15, the association unit 16, and the response time calculation unit 17 is realized by processing executed by the CPU 106 by a program installed in the HDD 108. The operation log table 21, the user operation table 22, and the drawing processing table 23 can be realized using, for example, the RAM 104, the HDD 108, or a storage device connected to the capture server 4 via the network 5.

  As described above, between the client device 1 and the service server 2 in each embodiment, the input operation data and the drawing process data are transmitted and received according to the communication protocol. The capture server 4 according to each embodiment calculates a shift of coordinate information between the input operation data and the drawing processing data, and calculates logical coordinate information using the shift. Based on the logical coordinate information, the actual “user operation” is associated with the “drawing process”. The capture server 4 measures the response time from the difference in time information between the input operation data and the drawing process data thus associated, and thereby can accurately measure the quality of the remote desktop system 10.

  As described above, the association method, the association program, and the association apparatus have been described in the above embodiments, but the present invention is not limited to the above embodiments, and various modifications and improvements can be made within the scope of the present invention. It is. Further, in the present invention, the techniques described in the above embodiments can be combined within a consistent range.

  For example, in the above-described embodiment, as the main input operation, “window movement” by “movement of the mouse cursor” on the screen based on the operation of the mouse or the keyboard by the user has been described as an example. However, the input operation is not limited to this, and includes an operation including coordinates according to various screen operations by the user. For example, the input operation may be “movement of some object” displayed on the screen or any other operation. The input operation includes not only an operation according to a screen operation directly by a user using a mouse or a keyboard but also an operation according to a screen operation remotely by a user's gesture operation.

  According to the associating method according to the present invention, even when the input operation data transmitted by the client device and the drawing processing data received by the client device from the service server are not necessarily acquired one-to-one, the performance and quality of the system can be improved. It can be evaluated more correctly.

  Further, the association method according to the present invention can calculate the response time for each user by including the user identification information in the input operation data and the drawing processing data.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
An association method executed by a computer capable of acquiring input operation data including coordinates according to a screen operation in a remote desktop system and drawing processing data including coordinates of a drawing processed area,
An input operation data group and a drawing process data group within a predetermined period are extracted from the acquired input operation data and the drawing process data,
Calculating a relative position between coordinates included in one input operation data of the input operation data group and coordinates of a drawing area included in one drawing processing data of the drawing processing data group;
Based on the relative position of the coordinates, calculate coordinates assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
Based on the assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
Association method including processing.
(Appendix 2)
The process of calculating the relative position is as follows:
Calculating a relative position between the coordinates included in the last input operation data in the input operation data group and the coordinates of the drawing area included in the last drawing process data in the drawing process data group;
The association method according to attachment 1.
(Appendix 3)
The process of calculating the relative position is as follows:
From the coordinates included in the last input operation data in the input operation data group and each of a plurality of coordinates demarcating the drawing area included in the last drawing process data in the drawing process data group, the coordinates Multiple relative positions of
The process of calculating the assumed coordinates is as follows:
Based on the relative positions of the plurality of coordinates, calculate a plurality of coordinates that are assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
The associating process is
Based on at least one of the plurality of assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
The association method according to attachment 2.
(Appendix 4)
An association program for causing an obtainable computer to execute input operation data including coordinates according to a screen operation in a remote desktop system and drawing processing data including coordinates of a drawing processed area,
An input operation data group and a drawing process data group within a predetermined period are extracted from the acquired input operation data and the drawing process data,
Calculating a relative position between coordinates included in one input operation data of the input operation data group and coordinates of a drawing area included in one drawing processing data of the drawing processing data group;
Based on the relative position of the coordinates, calculate coordinates assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
Based on the assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
An association program that includes processing.
(Appendix 5)
The process of calculating the relative position is as follows:
Calculating a relative position between the coordinates included in the last input operation data in the input operation data group and the coordinates of the drawing area included in the last drawing process data in the drawing process data group;
The association program according to attachment 4.
(Appendix 6)
The process of calculating the relative position is as follows:
From the coordinates included in the last input operation data in the input operation data group and each of a plurality of coordinates demarcating the drawing area included in the last drawing process data in the drawing process data group, the coordinates Multiple relative positions of
The process of calculating the assumed coordinates is as follows:
Based on the relative positions of the plurality of coordinates, calculate a plurality of coordinates that are assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
The associating process is
Based on at least one of the plurality of assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
The association program according to attachment 5.
(Appendix 7)
An association device capable of acquiring input operation data including coordinates according to a screen operation in a remote desktop system and drawing processing data including coordinates of a region subjected to drawing processing,
A data extraction unit that extracts an input operation data group and a drawing process data group within a predetermined period from the acquired input operation data and the drawing process data;
Relative position calculation for calculating a relative position between coordinates included in one input operation data of the input operation data group and coordinates of a drawing area included in one drawing processing data of the drawing processing data group And
An assumed position calculation unit for calculating coordinates assumed to be included in the input operation data from other drawing process data included in the drawing process data group based on the relative position of the coordinates;
An association unit that associates input operation data included in the input operation data group with the other drawing processing data based on the assumed coordinates;
An associating device.
(Appendix 8)
The relative position calculator is
Calculating a relative position between the coordinates included in the last input operation data in the input operation data group and the coordinates of the drawing area included in the last drawing process data in the drawing process data group;
The association device according to attachment 7.
(Appendix 9)
The relative position calculator is
From the coordinates included in the last input operation data in the input operation data group and each of a plurality of coordinates demarcating the drawing area included in the last drawing process data in the drawing process data group, the coordinates Multiple relative positions of
The assumed position calculation unit
Based on the relative positions of the plurality of coordinates, calculate a plurality of coordinates that are assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
The association unit
Based on at least one of the plurality of assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
The association device according to attachment 8.

1: Client device 2: Service server 3: Switch 4: Capture server 5: Network 10: Remote desktop system 11: Capture unit 12: L7 analysis unit 13: Data extraction unit 14: Relative position calculation unit 15: Assumed position calculation unit 16 : Association unit 17: Response time calculation unit 20: Storage unit 21: Operation log table 22: User operation table 23: Drawing process table 24: Response time table

Claims (5)

An association method executed by a computer capable of acquiring input operation data including coordinates according to a screen operation in a remote desktop system and drawing processing data including coordinates of a drawing processed area,
An input operation data group and a drawing process data group within a predetermined period are extracted from the acquired input operation data and the drawing process data,
Calculating a relative position between coordinates included in one input operation data of the input operation data group and coordinates of a drawing area included in one drawing processing data of the drawing processing data group;
Based on the relative position of the coordinates, calculate coordinates assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
Based on the assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
Association method including processing. The process of calculating the relative position is as follows:
Calculating a relative position between the coordinates included in the last input operation data in the input operation data group and the coordinates of the drawing area included in the last drawing process data in the drawing process data group;
The association method according to claim 1. The process of calculating the relative position is as follows:
From the coordinates included in the last input operation data in the input operation data group and each of a plurality of coordinates demarcating the drawing area included in the last drawing process data in the drawing process data group, the coordinates Multiple relative positions of
The process of calculating the assumed coordinates is as follows:
Based on the relative positions of the plurality of coordinates, calculate a plurality of coordinates that are assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
The associating process is
Based on at least one of the plurality of assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
The association method according to claim 2. An association program for causing an obtainable computer to execute input operation data including coordinates according to a screen operation in a remote desktop system and drawing processing data including coordinates of a drawing processed area,
An input operation data group and a drawing process data group within a predetermined period are extracted from the acquired input operation data and the drawing process data,
Calculating a relative position between coordinates included in one input operation data of the input operation data group and coordinates of a drawing area included in one drawing processing data of the drawing processing data group;
Based on the relative position of the coordinates, calculate coordinates assumed to be included in the input operation data from other drawing processing data included in the drawing processing data group,
Based on the assumed coordinates, the input operation data included in the input operation data group is associated with the other drawing processing data.
An association program that includes processing. An association device capable of acquiring input operation data including coordinates according to a screen operation in a remote desktop system and drawing processing data including coordinates of a region subjected to drawing processing,
A data extraction unit that extracts an input operation data group and a drawing process data group within a predetermined period from the acquired input operation data and the drawing process data;
Relative position calculation for calculating a relative position between coordinates included in one input operation data of the input operation data group and coordinates of a drawing area included in one drawing processing data of the drawing processing data group And
An assumed position calculation unit for calculating coordinates assumed to be included in the input operation data from other drawing process data included in the drawing process data group based on the relative position of the coordinates;
An association unit that associates input operation data included in the input operation data group with the other drawing processing data based on the assumed coordinates;
An associating device.

Images