WO2014080440A1 - Information processing device, control method, and control program - Google Patents

Abstract

In order to improve a client operation response in, for example, a thin-client system, an information processing device for displaying an execution result based on an operation instruction from a terminal device on the terminal device is equipped with: a determination unit for determining a processing method pertaining to the data quantity of the image data of the execution result, on the basis of an indicator pertaining to response time to the operation instruction from the terminal device; and a transmission unit for transmitting image data processed using the determined processing method to the terminal device.

Description

Information processing apparatus, control method, and control program

The present invention relates to an information processing apparatus, a control method, and a control program.

The thin client system is a system for causing a server to execute an application based on operation information from the client, for example, and displaying the execution result of the application on the client.

As described above, since the thin client system causes the server to execute the application, the client may have only a minimum function, for example, a communication function and an input function.

For this reason, thin client systems are beginning to spread in the engineering field using applications that process fine images, such as CAD (Computer-Aided Design), and applications that process moving images such as simulations.

However, for example, when the server processes a large amount of data such as a moving image, the response time until the execution result of the application resulting from the operation instruction of the client is actually displayed on the client may be affected.

Such a problem is common when large-capacity data transmission occurs in screen update between the client and server of the thin client system. This is not a problem that arises only when the server processes video.

For this reason, as a technique for improving the operation response of the client, in the area where the change frequency of the image data exceeds the threshold value, that is, in the high-frequency change area, the image data is compressed by the moving image compression method, Transmission techniques have been proposed.

JP 2011-238014 A

However, since the compression method for moving images uses a compression technique peculiar to moving images such as motion vectors, inter-frame prediction, and motion compensation, a huge amount of calculation is required as compared with the compression method for still images. For this reason, the response time until the operation instruction of the client is reflected on the display screen of the client cannot be sufficiently shortened.

The disclosed technology provides an information processing apparatus, an image transmission program, an image display program, and an image display method that improve an operation response of a client.

According to the disclosure of the present application, in the information processing apparatus that causes the terminal device to display an execution result based on the operation instruction from the terminal device, the execution is performed based on an index related to a response time to the operation instruction from the terminal device. An information processing apparatus is provided that includes a determination unit that determines a processing method related to the data amount of the resulting image data, and a transmission unit that transmits image data processed by the determined processing method to the terminal device. *

According to the present invention, the operational response of the client can be improved.

FIG. 1 is a schematic diagram of a hardware configuration of a server device according to an embodiment. FIG. 2 is a schematic diagram of a hardware configuration of the client terminal according to the embodiment. FIG. 3 is a schematic diagram of functional blocks of the thin client system according to the embodiment. FIG. 4 is a schematic diagram illustrating an example of division of image data according to an embodiment. FIG. 5 is a schematic diagram illustrating a procedure for determining the change frequency of image data according to an embodiment. FIG. 6 is a schematic diagram illustrating a correction procedure for a mesh assembly according to an embodiment. FIG. 7 is a schematic diagram illustrating a procedure for synthesizing a candidate for a frequently changed region according to an embodiment. FIG. 8 is a schematic diagram illustrating a notification procedure of attribute information of a frequently changed region according to an embodiment. FIG. 9 is a schematic diagram of a still image compression time table and a moving image compression table according to an embodiment. FIG. 10 is a flowchart of image transmission processing by the server device according to the embodiment. FIG. 11 is a flowchart of compression method determination processing by the server device according to the embodiment. FIG. 12 is a schematic diagram illustrating a specific example of compression method determination processing by the server device according to the embodiment. FIG. 13 is a flowchart of image display processing by the client terminal according to the embodiment.

Hereinafter, an embodiment will be described with reference to the drawings.

(Hardware configuration of server device 10)
FIG. 1 is a schematic diagram of a hardware configuration of a server device 10 according to an embodiment.

As shown in FIG. 1, the server apparatus 10 according to the present embodiment includes a CPU (Central Processing Unit) 101, a main memory 102, an auxiliary memory 103, an operation device 104, a display 105, and a communication module 106 as hardware modules. . These hardware modules are interconnected by a bus 107.

CPU 101 controls various hardware modules of server device 10. Furthermore, the CPU 101 executes various functions by reading various programs stored in the auxiliary memory 103 into the main memory 102 and executing the various programs read into the main memory 102. Details of the various functions will be described later.

The main memory 102 stores various programs executed by the CPU 101. Further, the main memory 102 is used as a work area for the CPU 101 and stores various data necessary for processing by the CPU 101. For example, a RAM (Random Access Memory) may be used as the main memory 102.

The auxiliary memory 103 stores various programs for operating the server device 10. Examples of the various programs include an application program such as a server control program 1000 executed by the server device 10 and an OS 1100 that is an execution environment of the application program. The server control program 1000 provides the server-side remote screen control unit 15 according to the present embodiment. As the auxiliary memory 103, for example, a non-volatile memory such as a hard disk or a flash memory may be used.

The communication module 106 receives operation information from the client terminal 20 and transmits image data and attribute information as an execution result of the application program to the client terminal 20. Further, the communication module 106 receives the data transfer rate from the server device 10 to the client terminal 20 calculated by the client terminal 20.

(Hardware configuration of client terminal 20)
FIG. 2 is a schematic diagram of a hardware configuration of the client terminal 20 according to the embodiment.

As shown in FIG. 2, the client terminal 20 according to the present embodiment includes a CPU 201, a main memory 202, an auxiliary memory 203, a display 204, a communication module 205, and an operation device 206 as hardware modules. These hardware modules are interconnected by a bus 207.

The CPU 201 controls various hardware modules of the client terminal 20. Furthermore, the CPU 201 executes various functions by reading various programs stored in the auxiliary memory 203 into the main memory 202 and executing the various programs read into the main memory 202. Details of the various functions will be described later.

The main memory 202 stores various programs executed by the CPU 201. Further, the main memory 202 is used as a work area for the CPU 201 and stores various data necessary for processing by the CPU 201. For example, a RAM or the like may be used as the main memory 202.

The auxiliary memory 203 stores various programs for operating the client terminal 20. Examples of the various programs include an application program such as a client control program 2000 executed by the client terminal 20 and an OS 2100 that is an execution environment of the application program. The client control program 2000 provides the client side remote screen control unit 23. As the auxiliary memory 203, for example, a nonvolatile memory such as a hard disk or a flash memory may be used.

The display 204 displays image data such as an execution result of the application program by the CPU 201 and an execution result of the application program by the server device 10, for example.

The communication module 205 transmits, for example, operation information received from the operation device 206 to the server device 10 and receives image data and attribute information generated by the server device 10. Further, the communication module 205 transmits the data transfer rate from the server device 10 to the client terminal 20 to the server device 10.

The operation device 206 acquires an operation instruction from the user of the client terminal 20. As the operation device 206, for example, a keyboard, a mouse, a pointing device, or the like may be used.

(Outline of thin client system 30)
FIG. 3 is a schematic diagram of functional blocks of the thin client system 30 according to the embodiment.

As shown in FIG. 3, the thin client system 30 according to the present embodiment includes a server device 10 and a client terminal 20, and controls the server device 10 to control a screen displayed by the client terminal 20. That is, the thin client system 30 operates the server device 10 based on an operation instruction from the client terminal 20 and causes the client terminal 20 to display an execution result of the application program by the server device 10 and retained data.

In the present embodiment, one client terminal 20 is connected to one server apparatus 10, but the number of client terminals 20 connected to the server apparatus 10 is not particularly limited.

The server device 10 and the client terminal 20 are connected via a network 40 so that they can communicate with each other. The network 40 is not limited to wired or wireless, and any type of communication network such as the Internet, LAN (Local Area Network), VPN (Virtual Private Network), or the like may be used. A communication protocol between the server device 10 and the client terminal 20 is not particularly limited, but for example, a VNC (Virtual Network Computing) RFB (Remote Frame Buffer) protocol may be used.

The server apparatus 10 is installed with an application program for remote screen control for the server, that is, a server control program 1000, and provides a service for remotely controlling a screen to be displayed on the client terminal 20.

The server control program 1000 provides a remote screen control service as a basic function. For example, when the server apparatus 10 acquires operation information from the client terminal 20, the server apparatus 10 causes an application program operating on the server apparatus 10 to execute a process requested by the operation information based on the server control program 1000. Further, the server device 10 generates screen information for displaying the execution result of the application program, that is, image data based on the server control program 1000, and transmits the image data to the client terminal 20. At this time, if there is image data that has been displayed on the client terminal 20 before the generation of the image data, the server device 10 changes the image data of the change area from the image data, the image data and attributes of the update rectangle in the present embodiment. Information is transmitted to the client terminal 20.

Furthermore, based on the server control program 1000, the server apparatus 10 converts image data in an area where the change frequency exceeds the frequency threshold among image data to be displayed on the client terminal 20, that is, image data in the high-frequency change area, And compressed to the client terminal 20. For example, the server device 10 divides the image data for displaying the execution result of the application program by the server device 10 into a plurality of regions, and the image data of the region where the change frequency exceeds the frequency threshold, that is, the high-frequency change region And attribute information is transmitted to the client terminal 20. As a compression method, for example, an MPEG method such as MPEG-2 (Moving Picture Experts Group IV-Exp2) or MPEG-4 (Moving Picture Experts Group IV- 4) may be used. However, the compression method according to the present invention is not limited to the data compression of the MPEG method, and any compression encoding method such as Motion-JPEG (Joint Photographic Experts Group) may be used as long as it is a compression method for moving images. May be used.

The client terminal 20 is installed with an application program for remote screen control for clients, that is, a client control program 2000, and enjoys the provision of a remote screen control service by the server device 10. As the client terminal 20, a fixed terminal such as a personal computer, or a mobile terminal such as a mobile phone, a PHS (Personal Handyphone System), or a PDA (Personal Digital Assistant) may be used.

The client terminal 20 notifies the server device 10 of the operation information received by the operation device 206 based on the client control program 2000. For example, the client terminal 20 sends the mouse click, double click, drag, mouse cursor movement direction and movement amount, mouse wheel rotation amount, keyboard pressing key type, and the like as operation information to the server device 10. Notice.

Further, the client terminal 20 displays the image data transmitted from the server device 10 on the display 204 of the client terminal 20 based on the client control program 2000. For example, when the update rectangle image data and attribute information are received from the server device 10, the client terminal 20 displays the update rectangle image data at the position specified by the attribute information in the size specified by the attribute information. . Further, when the image data and attribute information of the frequently changed area are received from the server device 10, the client terminal 20 designates the image data of the frequently changed area in the position designated by the attribute information by the attribute information. Display in size.

(Detailed functions of server device 10)
As illustrated in FIG. 3, the server device 10 according to the present embodiment includes an OS execution control unit 11, an application execution control unit 12, a graphic driver 13, a frame buffer 14, and a server-side remote screen control unit 15.

The OS execution control unit 11, the application execution control unit 12, the graphic driver 13, and the frame buffer 14 are all realized by reading the OS 1100 into the main memory 102 and executing the OS 1100 read into the main memory 102.

The server-side remote screen control unit 15 is realized by the CPU 101 reading the server control program 1000 into the main memory 102 and executing the server control program 1000 read into the main memory.

[OS execution control unit 11]
The OS execution control unit 11 controls execution of the OS 1100. For example, the OS execution control unit 11 detects an application program activation instruction and a command to the application program from the operation information acquired by the operation information acquisition unit 151. Specifically, when a request operation for starting an application program is detected, the OS execution control unit 11 instructs the application execution control unit 12 to start the application program associated with the icon. When a request operation for command execution to the application program is detected, the OS execution control unit 11 instructs the application execution control unit 12 to execute the command.

[Application execution control unit 12]
The application execution control unit 12 controls the execution of the application program based on an instruction from the OS execution control unit 11. The instruction from the OS execution control unit 11 is an application program activation instruction or an application program command execution instruction. Further, the application execution control unit 12 instructs the graphic driver 13 to draw the display image of the execution result by the application program in the frame buffer 14. At this time, the application execution control unit 12 also instructs the graphic driver 13 to draw the display image.

[Graphic driver 13]
The graphic driver 13 draws a display image in the frame buffer 14 based on an instruction from the application execution control unit 12. Specifically, when a drawing instruction is received from the application execution control unit 12, the graphic driver 13 bitmaps the display image of the execution result by the application program to the drawing position specified by the application program in the frame buffer 14. Draw in format.

[Frame buffer 14]
The frame buffer 14 stores bitmap-format image data drawn by the graphic driver 13. As the frame buffer 14, for example, a random access memory (RAM) such as a video random access memory (VRAM), a read only memory (ROM), a semiconductor memory element such as a flash memory, or a storage device such as a hard disk or an optical disk is used. May be.

[Server-side remote screen control unit 15]
The server-side remote screen control unit 15 includes an operation information acquisition unit 151, an image generation unit 152, a change frequency determination unit 153, a high frequency change region identification unit 154, an encoder 155, an image transmission unit 156, and a compression method determination unit 157. .

(Operation Information Acquisition Unit 151)
The operation information acquisition unit 151 acquires operation information from the client terminal 20. As the operation information, for example, mouse click, double click, drag, mouse cursor movement direction and movement amount, mouse wheel rotation amount, keyboard pressing key type, and the like may be used.

(Image generation unit 152)
The image generation unit 152 generates image data to be displayed on the display 204 of the client terminal 20. Specifically, when bitmap format image data is stored in the frame buffer 14, the image generation unit 152 obtains image data of the immediately previous frame (previous frame) and image data of the latest frame (current frame). Compare. The image data of the immediately previous frame is the image data that was displayed on the client terminal 20 immediately before, and the image data of the latest frame is the image data that is newly stored in the frame buffer 14. Furthermore, the image generation unit 152 generates a rectangular update region that includes a change pixel from the image data of the immediately preceding frame, that is, an update rectangle. The image generation unit 152 generates an update rectangle transmission packet in order to transfer the update rectangle image data to the client terminal 20. The update rectangle transmission packet includes the update rectangle image data compressed by the still image compression method, the attribute information specifying the display position and display size of the image data, the packet transmission time by the server device 10, and the like. .

(Change frequency determination unit 153)
The change frequency determination unit 153 divides the image data drawn in the frame buffer 14 into a plurality of regions, and determines the change frequency between frames for each region. For example, the change frequency determination unit 153 accumulates the update rectangle attribute information generated by the image generation unit 152 in the work memory for a predetermined period. As the attribute information, information specifying the position and size of the update rectangle, for example, the coordinates of the upper left vertex of the update rectangle, and the width and height of the update rectangle may be used. The accumulation period for accumulating the update rectangle attribute information is not particularly limited, but may be one second, for example. When a predetermined period has elapsed since the start of accumulation of the update rectangle image data, the change frequency determination unit 153 uses the change frequency determination map obtained by dividing the image data to be displayed on the client terminal 20 into a mesh shape, thereby changing the change frequency. Is determined.

FIG. 4 is a schematic diagram illustrating an example of division of image data according to an embodiment.

As shown in FIG. 4, the change frequency determination map 50 is divided into a plurality of meshes 51. Each mesh 51 includes a plurality of pixels 52. In the present embodiment, each mesh 51 includes 8 × 8 (64 pixels) 52.

Here, the change frequency determination unit 153 develops the update rectangle on the change frequency determination map 50 based on the update rectangle attribute information stored in the work memory, that is, the position and size of the update rectangle. Furthermore, every time the update rectangle is developed on the change frequency determination map, the change frequency determination unit 153 adds the number of changes of the mesh 51 where the update rectangles overlap even a little.

FIG. 5 is a schematic diagram for explaining a procedure for determining the change frequency of image data according to an embodiment.

As shown in FIG. 5A, when the update rectangle A1 is developed in the change frequency determination map 50, the update rectangle A1 overlaps the mesh 51 (9 pieces) in the shaded portion. For this reason, the change frequency determination unit 153 adds the number of updates of the mesh 51 in the shaded portion one by one. In this embodiment, since the initial value of the number of changes of each mesh 51 is set to 0, as a result of the development of the update rectangle A1, the number of changes of the shaded portion is changed from “0” to “ 1 ".

Next, as shown in FIG. 5B, when the update rectangle A2 is developed in the change frequency determination map 50, the update rectangle A2 overlaps with the meshes 51 (six pieces) in the shaded portion. For this reason, the change frequency determination unit 153 adds the number of updates of the mesh 51 in the shaded portion one by one. At this time, since the number of changes of the mesh 51 in the shaded portion is 1, the number of changes in the shaded portion is changed from “1” to “1” as described in each mesh 51 as a result of the development of the update rectangle A2. 2 ”.

Thus, each time the update rectangles A1, A2,... Are developed in the change frequency determination map 50, the number of changes of each mesh 51 is updated. When all the update rectangles accumulated in the work memory are expanded in the change frequency determination map 50, the change frequency determination unit 153 calculates the number of changes per predetermined period, that is, the change frequency, and the change frequency is increased. A mesh 51 exceeding the threshold is acquired. For example, when the frequency threshold value is “4”, the mesh 51 of the shaded portion is acquired as shown in FIG. The frequency threshold value may be a numerical value set by the developer of the server control program 1000 or a numerical value directly input by the end user.

(High-frequency change area identification unit 154)
The high-frequency change area identifying unit 154 identifies “high-frequency change areas” from areas where the change frequency of the image data to be displayed on the client terminal 20 exceeds the threshold value. Specifically, when the change frequency determination unit 153 acquires a mesh whose number of changes exceeds the frequency threshold, the high frequency change region identification unit 154 includes a plurality of meshes acquired by the change frequency determination unit 153. Then, meshes connected to each other (adjacent) are connected to generate a so-called mesh connected body. At this time, a mesh may be added to the mesh connection body and corrected to the minimum rectangle including the mesh connection body. In this embodiment, a mesh connected body corrected to the minimum rectangle is used.

FIG. 6 is a schematic diagram illustrating a correction procedure for a mesh assembly according to an embodiment.

As shown in FIG. 6A, when a missing region 54 exists in the mesh connected body 53, an interpolation mesh 55 for interpolating the lost region 54 is added to the mesh connected body 53 as shown in FIG. In addition, a new mesh connected body 56 corrected to a minimum rectangle including the mesh connected body 53 is generated. The new mesh connected body 56 becomes a candidate for the frequently changed region.

Further, when the high frequency change area identifying unit 154 detects a plurality of new mesh connected bodies 56, that is, a plurality of high frequency change area candidates, the interval (distance) d of the high frequency change area candidates is determined in advance. Determine if it is less than the given distance threshold. When there is a candidate for the frequently changed area whose interval (distance) d between the candidates for the frequently changed area is smaller than the distance threshold, the frequently changed area identifying unit 154 includes a minimum that includes the candidates for the frequently changed area. Generate a rectangle.

FIG. 7 is a schematic diagram illustrating a procedure for synthesizing a candidate for a frequently changed region according to an embodiment.

As shown in FIG. 7A, when the distance (distance) d between the candidates B1 and B2 of the frequently changed region is smaller than a predetermined distance threshold, as shown in FIG. The high-frequency change area identifying unit 154 adds the complement area C to the high-frequency change area candidates B1 and B2, and generates a minimum rectangle that includes both the high-frequency change area candidates B1 and B2. The high-frequency change area identifying unit 154 identifies the minimum rectangle that includes both the high-frequency change area candidates B1 and B2 as the high-frequency change area.

However, when the interval (distance) d between the frequently changed regions B1 and B2 is larger than the distance threshold, the frequently changed region identifying unit 154 changes each of the frequently changed region candidates B1 and B2 to the frequently changed region. Identify as a region.

When the frequently changed area is identified, the frequently changed area identifying unit 154 transmits the attribute information of the frequently changed area, that is, the position and size of the frequently changed area to the client terminal 20.

FIG. 8 is a schematic diagram for explaining a notification procedure of attribute information of a frequently changed region according to an embodiment. FIG. 8A is a display image to be displayed on the client terminal 20, and FIGS. These show a change frequency determination map for determining the change frequency of the image data drawn in the frame buffer 14.

As shown in FIG. 8A, the display image displayed on the client terminal 20 includes a mouse cursor 61, a browser screen 62, and a moving image playback screen 63.

Here, the display image changes with time, and as shown in FIG. 8B, an update rectangle 64 resulting from the movement of the mouse cursor 61 and an update rectangle 65 resulting from the change in the moving image playback screen 63 , Is acquired. However, since the browser screen 62 is a still image, an update rectangle resulting from the browser screen 62 is not detected.

From the above situation, when the display image changes over time and the frequently changed area 66 is acquired, the frequently changed area identifying unit 154, as shown in FIG. The attribute information of the change area 66, that is, the coordinates (x, y) of the upper left vertex of the high frequency change area 66 and the width w and height h of the high frequency change area 66 are transmitted to the client terminal 20.

The high-frequency change area identifying unit 154 determines whether or not the update rectangle is included in the high-frequency change area stored in the work memory, that is, the moving image reproduction area, every time an update rectangle is generated by the image generation unit 152. When the update rectangle is not included in the high-frequency change area, the high-frequency change area identification unit 154 instructs the image transmission unit 156 to transmit the image data and attribute information of the update rectangle. When the update rectangle is included in the high-frequency change area, the high-frequency change area identification unit 154 does not instruct the image transmission unit 156 to transmit the image data and attribute information of the update rectangle. When the update rectangle is caused by the movement of the mouse cursor drawn by the OS execution control unit 11, the image data and attribute information of the update rectangle caused by the mouse cursor may be transmitted exceptionally.

Further, the frequently changed area identifying unit 154 determines whether or not the attribute information of the frequently changed area is registered in the work memory every time the bitmap format image data is drawn in the frame buffer 14. When the attribute information of the frequently changed area is registered, the frequently changed area identifying unit 154 acquires the image data of the frequently changed area among the image data drawn in the frame buffer 14.

(Encoder 155)
The encoder 155 compresses the update rectangle image data generated by the image generation unit 152 or the image data of the high-frequency change area acquired by the high-frequency change area identification unit 154 using a compression method for still images or moving images, respectively. (Encode). For example, when compressing image data in the frequently changed area, the encoder 155 starts encoding the image data in the frequently changed area when the image data in the frequently changed area reaches the number of frames that can be streamed. As a compression method for moving images, for example, an MPEG method such as MPEG-2 or MPEG-4 or a Motion-JPEG method may be used. Further, the encoder 155 may update the still image compression time table Ta or the moving image compression time table Tb based on the compression history by the still image or moving image compression method. For example, each time image data in an update rectangle or frequently changed area is compressed by a still image or moving image compression method, the relationship between the number of pixels in the update rectangle or frequently changed area and the compression time is recorded. When a plurality of relationships are acquired, the still image compression time table Ta or the moving image compression time table Tb is updated.

(Image transmission unit 156)
The image transmission unit 156 transmits the update rectangle image data and attribute information compressed by the encoder 155 to the client terminal 20. When transmitting update rectangle image data, for example, an RFB protocol in VNC may be used as a communication protocol. Further, the image transmission unit 156 transmits the image data and attribute information of the frequently changed region compressed by the encoder 155 to the client terminal 20. When transmitting the image data of the frequently changed area, for example, RTP (Real-time Transport Protocol) may be used as a communication protocol.

(Compression method determination unit 157)
The compression method determination unit 157 includes a transfer rate acquisition unit 1571, a processing order determination unit 1572, a response time estimation unit 1573, a compression method selection unit 1574, and a processing time storage unit 1575.

The transfer rate acquisition unit 1571 acquires the data transfer rate calculated by the client terminal 20. The data transfer rate is a data amount per unit time of data transferred from the server device 10 to the client terminal 20.

The processing order determination unit 1572 assigns the processing order of the compression method determination process to each of the high frequency change areas identified by the high frequency change area identification unit 154. Specifically, the processing order determination unit 1572 assigns the processing order in descending order of the area (number of pixels) of the frequently changed region.

The response time estimation unit 1573 is based on the data transfer rate acquired by the transfer rate acquisition unit 1571 and the still image compression time table Ta or the moving image compression table Tb stored in the processing time storage unit 1575. The time required from the start of image data compression to the acquisition of image data by the client terminal 20 is calculated. In the present embodiment, the required time calculated here is the response time. The response time is an example of an index related to response time.

Usually, the response time is the time from when the client terminal 20 receives an operation instruction from the operation device 206 to when the execution result of the application program resulting from the operation instruction is reflected on the display screen of the client terminal 20. However, the time until the server device 10 acquires the operation information received by the client terminal 20 is not affected by the size of the image data, and is considerably smaller than the compression time of the image data. Then, the time excluding this is set as the response time.

Based on the estimated response time calculated by the response time estimation unit 1573, the compression method selection unit 1574 selects a compression method for image data from a compression method for still images or a compression method for moving images for each high-frequency change area. To do.

FIG. 9 is a schematic diagram of a still image compression table Ta and a moving image compression time table Tb according to an embodiment.

As shown in FIG. 9A, the still image compression time table Ta records the relationship between the number of pixels (area) and the processing time when image data is compressed by a still image compression method. As shown in FIG. 9B, the moving image compression time table Tb stores the relationship between the number of pixels (area) and the processing time when image data is compressed by a moving image compression method.

Comparing the still image compression time table Ta and the moving image compression time table Tb, it can be seen that the compression time by the compression method for moving images is significantly longer than the compression time by the compression method for still images. The still image compression time table Ta and the moving image compression time table Tb are updated every time the encoder 155 compresses the image data.

(Detailed functions of client terminal 20)
As illustrated in FIG. 3, the client terminal 20 according to the present embodiment includes an operation information receiving unit 21, a display unit 22, and a client-side remote screen control unit 23.

Both the operation information reception unit 21 and the display unit 22 are realized by the CPU 201 reading the OS 2100 into the main memory 202 and executing the OS 2100 read into the memory 202.

The client-side remote screen control unit 23 is realized by the CPU 201 reading the client control program 2000 into the main memory 202 and executing the client control program 2000 read into the main memory 202.

The operation information reception unit 21 receives various information acquired from the operation device 206, for example, an instruction input to the client-side remote screen control unit 23.

The display unit 22 causes the display 204 to display the update rectangle or the image data of the frequently changed region transmitted from the server device 10 based on the instruction from the display control unit 234. At this time, the display unit 22 determines the display position and display size of the image data based on the position and size included in the attribute information.

[Client-side remote screen controller 23]
The client-side remote screen control unit 23 includes an operation information notification unit 231, an image reception unit 232, a decoder 233, a display control unit 234, and a transfer rate calculation unit 235.

The operation information notifying unit 231 notifies the server device 10 of the operation information received by the operation information receiving unit 21. For example, the operation information notification unit 231 uses, for example, the left and right mouse clicks, double clicks, drags, the movement direction and movement amount of the mouse cursor, the rotation amount of the mouse hole, the type of the key pressed on the keyboard as the operation information as the server device. 10 is notified.

The image receiving unit 232 receives the update rectangle or the frequently changed area image data and attribute information transmitted from the server device 10. The attribute information includes the position and size of the update rectangle or the frequently changed area. Since the image data and the attribute information are packetized, the image receiving unit 232 extracts the image data and the attribute information from the packet.

The decoder 233 restores (decodes) the update rectangle or the frequently changed area image data received by the image receiving unit 232 by a designated compression method, that is, a still image compression method or a moving image compression method. Note that the decoder 233 according to the present embodiment restores image data using a designated compression method, but the present invention is not limited to this. For example, a dedicated decoder may be prepared for each compression method.

The display control unit 234 instructs the display unit 22 to display the image data restored by the decoder 233 based on the attribute information received by the image receiving unit 232. Specifically, the display control unit 234 notifies the display unit 22 of an instruction to display image data having a size specified by the attribute information at a position specified by the attribute information.

The transfer rate calculation unit 235 analyzes the packet of the image data received by the image reception unit 232 and calculates the data transfer rate from the server device 10 to the client terminal 20. Specifically, the transfer rate calculation unit 235 obtains the packet transmission time by the server device 10 from the image data packet, and the difference between the packet reception time by the client terminal 20 and the packet transmission time by the server device 10; That is, the data transfer time is calculated. Further, the transfer rate calculation unit 235 obtains the data amount of the image data based on the packet received from the server device 10 and divides this by the data transfer time, so that the data transfer amount per unit time, that is, the data Calculate the transfer rate.

[Image Transmission Processing by Server Device 10]
FIG. 10 is a flowchart of image transmission processing by the server device 10 according to the embodiment.

When the image data is drawn in the frame buffer 14, as shown in FIG. 10, the image generation unit 152 generates an update rectangle that includes the changed portion from the image data displayed on the client terminal 20 immediately before (step S101). ).

Next, the image generation unit 152 generates a packet for transmitting the update rectangle image data based on the update rectangle image data, that is, the image data of the region included in the update rectangle (step S102).

Next, the change frequency determination unit 153 stores the update rectangle generated by the image generation unit 152 in the work memory (step S103).

Next, the change frequency determination unit 153 determines whether or not the elapsed time from the start of accumulation of the update rectangle has exceeded a predetermined time threshold (step S104).

Here, if it is not determined that the elapsed time from the start of accumulation of the update rectangle has exceeded the time threshold value (No in step S104), the frequent change area identification unit 154 determines that the update rectangle generated by the image generation unit 152 Then, it is determined whether or not it is included in the high frequency change area stored in the work memory (step S115).

On the other hand, when it is determined that the elapsed time from the start of accumulation of the update rectangle has exceeded the time threshold value (Yes in step S104), the change frequency determination unit 153 determines the position and size of each update rectangle accumulated in the work memory. Based on the above, the update rectangle is developed on the change frequency determination map (step S105).

Next, the change frequency determination unit 153 acquires a mesh whose change frequency exceeds a predetermined frequency threshold, that is, a high-frequency mesh among the meshes of the change frequency determination map (step S106).

Next, the high-frequency change area identifying unit 154 determines whether there is a high-frequency mesh whose change frequency exceeds the frequency threshold (step S107).

Here, if it is not determined that there is a high-frequency mesh whose change frequency exceeds the frequency threshold (No in step S107), the high-frequency change area identifying unit 154 determines that there is no high-frequency change area, and The number of changes for each mesh of the map for change frequency determination stored in the memory is cleared (step S114).

On the other hand, if it is determined that there is a high-frequency mesh whose change frequency exceeds the frequency threshold value (Yes in step S107), the high-frequency change region identifying unit 154 connects the high-frequency mesh to generate a mesh connected body. (Step S108). At this time, the mesh connected body may be corrected to a minimum rectangle including the mesh connected body. In this embodiment, a mesh connected body corrected to the minimum rectangle is used. When only one high-frequency mesh exists, the high-frequency change area identifying unit 154 uses the high-frequency mesh as a mesh connected body.

Next, the high-frequency change area identification unit 154 determines whether or not there are a plurality of mesh connected bodies in the change frequency determination map (step S109).

Here, when it is determined that there are a plurality of mesh connected bodies (Yes in step S109), the high-frequency change area identifying unit 154 determines that the mutual distance is smaller than a predetermined distance threshold value. It is determined whether a body exists (step S110).

Here, when it is determined that there is a mesh connected body whose mutual distance is smaller than the distance threshold (Yes in step S110), the high-frequency change area identifying unit 154 combines these mesh connected bodies (step S110). S111). Further, the high-frequency change area identifying unit 154 identifies an area where a plurality of mesh connected bodies are combined as a high-frequency change area. At this time, a minimum rectangle including a plurality of mesh connected bodies may be used as the high frequency change region.

Next, the high-frequency change area identification unit 154 determines whether the area of the high-frequency change area is smaller than a predetermined area threshold (step S112).

On the other hand, when it is not determined that there are a plurality of mesh connected bodies (No in step S109), that is, when there is only one mesh connected body, the high-frequency change area identifying unit 154 displays the mesh connected body as a high-frequency change area. Further, it is determined whether or not the area of the frequently changed region is smaller than the area threshold (step S112).

Furthermore, when it is not determined that there is a mesh connected body whose mutual distance is smaller than the distance threshold (No in step S110), that is, although there are a plurality of mesh connected bodies, the mutual distance is less than the distance threshold value. Is also larger, the frequent change region identifying unit 154 identifies each mesh connected body as a frequent change region, and further determines whether the area of the frequent change region is smaller than the area threshold (step). S112).

Here, when it is determined that the area of the frequently changed region is smaller than the area threshold value (Yes in step S112), the image transmitting unit 156 transmits the attribute information of the frequently changed region to the client terminal 20 ( Step S113).

On the other hand, if it is not determined that the area of the frequently changed region is smaller than the area threshold value (No in step S112), that is, it is determined that the area of the frequently changed region is the same as or larger than the area threshold value. If so, the compression method determination unit 157 performs “compression method determination processing” (step S118). However, when the high-frequency change region combines a plurality of mesh connected bodies to generate the high-frequency change region, the compression method determination unit 157 determines each mesh connected body (high frequency change region candidate) before combining. Is re-identified as a frequently changed region. Details of the “compression method determination process” will be described later.

Next, the high-frequency change area identifying unit 154 clears the number of changes for each mesh of the change frequency determination map accumulated in the work memory (step S114).

Next, the high-frequency change area identifying unit 154 includes the update rectangle generated by the image generation unit 152 in the high-frequency change area stored in the work memory, that is, the area transmitted as a moving image by the image transmission unit 156. Is determined (step S115).

Here, when it is not determined that the update rectangle is included in the high-frequency change area (No in step S115), the encoder 155 compresses the image data of the update rectangle using the still image compression method (step S116).

Next, the image transmission unit 156 transmits the updated rectangular image data and attribute information compressed by the still image compression method to the client terminal 20 (step S117).

On the other hand, when it is determined that the update rectangle is included in the high frequency change area (Yes in step S115), the encoder 155 selects the image data of the high frequency change area, that is, among the image data drawn in the frame buffer 14, The image data in the region included in the high-frequency change region is compressed by the moving image compression method (step S119).

Next, the image transmission unit 156 transmits the image data of the frequently changed region compressed by the moving image compression method to the client terminal 20 (step S120). Note that the attribute information of the frequently changed region is transmitted to the client terminal 20 in advance.

[Compression method determination processing by server device 10]
FIG. 11 is a flowchart of compression method determination processing by the server device 10 according to the embodiment.

As illustrated in FIG. 11, the response time estimation unit 1573 is configured to perform the frequent change of the processing target from the frequently changed region identified by the frequently changed region identifying unit 154 based on the processing order determined by the processing order determining unit 1572. A region is selected (step S201).

Next, the response time estimation unit 1573 determines whether the area (number of pixels) of the frequently changed region to be processed is smaller than a predetermined area threshold (number of pixels) (step S202). .

Here, when it is determined that the area of the high-frequency change area to be processed is smaller than the area threshold (Yes in step S202), the response time estimation unit 1573 selects the high-frequency change area to be processed as a moving image area. The estimated response time when compressed by the compression method is calculated (step S203). The estimated response time here is the sum of the compression time of the image data in the high-frequency change area to be processed and the transfer time of the compressed image data from the server device 10 to the client terminal 20.

Specifically, the response time estimation unit 1573 determines the processing target high frequency change region based on the area of the processing target high frequency change region and the moving image compression time table Tb stored in the processing time storage unit 1575. A processing time for compression by the compression method for moving images, that is, a moving image compression time is acquired.

For example, as shown in FIG. 9B, according to the moving image compression time table Tb, when the number of pixels of the high-frequency change area to be processed is 155000 [pixel], the response time estimation unit 1573 40 [msec] is acquired as the moving image compression time of the image data in the high-frequency change area.

Further, the response time estimation unit 1573 calculates the data transfer time when the data amount of the image data compressed by the moving image compression method is transferred at the data transfer rate acquired from the transfer rate acquisition unit 1571.

Subsequently, the response time estimation unit 1573 adds the moving image compression time and the data transfer time to calculate the estimated response time of the frequently changed region to be processed.

Next, the compression method selection unit 1574 adds the newly calculated estimated response time to the estimated response time accumulated value stored in the work memory, and updates the estimated response time accumulated value (step S204). ). When the processing order of the high-frequency change area to be processed is No. 1, since the integrated value of the estimated response time is not stored in the work memory, the newly calculated estimated response time is used as the estimated response time. It is updated as the integrated value.

Next, the compression method selection unit 1574 determines whether or not the integrated value of the estimated response time is smaller than a predetermined time threshold value (step S205). The time threshold here is set to the minimum time during which the user of the client terminal 20 recognizes the response delay to the operation instruction, which is 150 [msec] in this embodiment.

Here, when it is determined that the integrated value of the estimated response time is smaller than the time threshold value (Yes in step S205), the compression method selection unit 1574 determines that the frequent change is identified by the frequent change region identifying unit 154. It is determined whether or not there is a high-frequency change area that has not been selected as a processing target so far, that is, an unselected high-frequency change area (step S206).

Here, when it is determined that there is an unselected high-frequency change area (Yes in step S206), the response time estimation unit 1573 again determines the unselected high-frequency change identified by the high-frequency change area identification unit 154. A high-frequency change area to be processed is selected from the area (step S201).

On the other hand, when it is not determined that there is an unselected frequently changed region (No in step S206), the compression method selecting unit 1574 is a compression method for all the frequently changed regions identified by the frequently changed region identifying unit 154. Then, a compression method for moving images is selected (step S207).

Next, the encoder 155 compresses the image data of all the frequently changed areas identified by the frequently changed area identifying unit 154 by the compression method selected by the compression method selecting unit 1574, that is, the moving image compression method. (Step S208).

Next, the image transmission unit 156 transmits the image data of the frequently changed region compressed by the encoder 155 to the client terminal 20 (step S209). At this time, the image transmission unit 156 also transmits attribute information of each frequently changed region to the client terminal 20.

On the other hand, when it is not determined that the integrated value of the estimated response time is smaller than the time threshold value (No in step S205), that is, when the integrated value of the estimated response time is equal to or greater than the time threshold value, the response time estimation is performed. The unit 1573 calculates an estimated response time when the image data of the high-frequency change area to be processed and the image data of the unselected high-frequency change area are compressed by the still image compression method (step S210). .

Specifically, the response time estimation unit 1573 includes the area (number of pixels) of the high-frequency change region to be processed and the unselected high-frequency change region, and the still image compression time table Ta stored in the processing time storage unit 1575. The processing time for compressing the image data of the high-frequency change area to be processed and the image data of the non-selected high-frequency change area by the compression method for still images, that is, the still image compression time, , Get each.

For example, as shown in FIG. 9A, according to the still image compression time table Ta, when the number of pixels in the high-frequency change area to be processed is 155000 [pixel], the response time estimation unit 1573 10 [msec] is acquired as the still image compression time of the image data in the frequently changed area. Note that the response time estimation unit 1573 similarly acquires the still image compression time of the image data of the unselected high frequency change area.

Furthermore, the response time estimation unit 1573 calculates the data transfer time when the data amount of the image data compressed by the still image compression method is transferred at the data transfer rate acquired from the transfer rate acquisition unit 1571.

Subsequently, the response time estimation unit 1573 generates a still image compression time for the image data in the high-frequency change area to be processed, a still image compression time for the image data in the unselected high-frequency change area, and a high-frequency change in the process target. The estimated response time is calculated by adding the data transfer time of the image data compressed by the still image compression method in the region and the unselected high-frequency change region.

Next, the compression method selection unit 1574 adds the newly calculated estimated response time to the accumulated value of the estimated response time stored in the work memory, and updates the accumulated value of the estimated response time (step S211). ).

Next, the compression method selection unit 1574 determines whether or not the integrated value of the estimated response time is smaller than a predetermined time threshold value (step S212).

Here, when it is determined that the integrated value of the estimated response time is smaller than the time threshold value (Yes in step S212), the compression method selection unit 1574, except for the high-frequency change region to be processed, As the compression method for the image data in the frequency change area, a compression method for moving images is selected, and the compression method for the image data in the high-frequency change area to be processed and the image data in the unselected high-frequency change area is static. A compression method for the image is selected (step S213).

Next, the encoder 155 compresses all the high-frequency change areas identified by the high-frequency change area identification unit 154 by the compression method (for still image / moving image) selected by the compression method selection unit 1574 ( Step S208).

Next, the image transmission unit 156 transmits the image data of the frequently changed area compressed by the still image / moving image compression method to the client terminal 20 (step S209).

[Specific example of compression method decision processing]
FIG. 12 is a schematic diagram illustrating a specific example of compression method determination processing by the server device 10 according to the embodiment.

As shown in FIG. 12A, when the area of the frequently changed region R exceeds the area threshold value (No in step S112), the compression method determination unit 157 starts the compression method determination process.

That is, when the area of the high-frequency change region R exceeds the area threshold, the moving image compression time Trd for compressing the image data of the high-frequency change region R by the compression method for moving images increases, and the estimated response time The time threshold is expected to be exceeded. For this reason, a compression method determination process is performed to shorten the response time in advance.

In the compression method determination process, mesh connected bodies M1 and M2 (candidates for high-frequency change areas) included in the high-frequency change area R are re-identified as high-frequency change areas R1 and R2, and the high-frequency change area R1 , R2 are calculated by the moving image compression times T1d and T2d for compressing them by the moving image compression method.

As shown in FIG. 12 (a), the target area to be compressed by the compression method for moving images is changed (reduced) from the high-frequency change area R to the high-frequency connectors R1 and R2. As shown, the total value of the moving image compression times T1d and T2d is shorter than the moving image compression time Trd for compressing the image data in the high-frequency change region R by the compression method for moving images.

However, if the transfer time Tdd for transferring the image data of the high-frequency change areas R1 and R2 compressed by the moving image compression method is added to the moving image compression times T1d and T2d, the estimated response time is set to the time threshold value. Exceed. Therefore, in order to further reduce the response time, the compression method determination unit 157 calculates a still image compression time T2s for compressing the image data in the high-frequency change area R2 by the still image compression method. Since the compression method of the high-frequency change area R2 is changed from the compression method for moving images to the compression method for still images, the calculation amount of the compression process is reduced. Therefore, the still image compression time T2s is longer than the moving image compression time T2d. Significantly shortened. For this reason, as shown in FIG. 12B, the total value of the moving image compression time T1d and the still image compression time T2s is further reduced, and the transfer time Tds is added to the moving image compression time T1d and the still image compression time T2s. However, the estimated response time does not exceed the time threshold.

Note that since the compression method for still images has a low compression rate, the amount of image data after compression is larger than the compression method for moving images. For this reason, if a still image compression method is used, the transfer time of image data increases. However, when the communication band is sufficiently secured, the estimated response time can be shortened because the decrease in the compression time is greater than the increase in the transfer time.

Therefore, the compression method determination unit 157 selects the moving image compression method as the compression method for the image data in the high-frequency change region R1, and also selects the still image compression method as the compression method for the image data in the high-frequency change region R2. select.

In the present embodiment, not all the high-frequency change areas are compressed as still images, but the high-frequency change areas that are compressed by the still image compression method in a range in which the estimated response time does not exceed the time threshold. Keep to a minimum. For this reason, an increase in the amount of data transferred from the server device 10 to the client terminal 20 can be suppressed.

[Image Display Processing by Client Terminal 20]
FIG. 13 is a flowchart of image display processing by the client terminal 20 according to the embodiment.

As shown in FIG. 13, first, the operation information receiving unit 21 receives operation information input by, for example, a mouse or a keyboard (step S301).

Next, the operation information notifying unit 231 transmits the operation information received by the operation information receiving unit 21 to the server device 10 (step S302).

Next, the image receiving unit 232 determines whether data has been received from the server device 10 (step S303).

Here, when it is determined that data has been received from the server device 10 (Yes in step S303), it is determined whether the data from the server device 10 is moving image data (step S304).

Here, when it is determined that the data from the server device 10 is moving image data (Yes in step S304), the decoder 233 restores the moving image data from the server device 10 by a moving image compression method (step S305). .

On the other hand, when it is not determined that the data from the server device 10 is moving image data (No in step S304), that is, when it is determined that the data from the server device 10 is still image data, the decoder 233 Still image data is restored by a still image compression method (step S306).

Next, the display control unit 234 controls the display unit 22 based on the attribute information transmitted from the server device 10, and the image data (still image data / moving image data) restored by the decoder 233 is transmitted to the client terminal 20. Are displayed on the display 204 (step S307).

Next, the transfer rate calculation unit 235 determines the difference between the transmission time of the packet of the image data (still image data / moving image data) received by the image reception unit 232 and the reception time when the client terminal 20 receives the packet. Then, based on the data amount of the image data, a data transfer amount per unit time, that is, a data transfer rate is calculated (step S308).

Next, the transfer rate calculation unit 235 notifies the server device 10 of the data transfer rate (step S309).

Next, the operation information reception unit 21 waits until operation information from, for example, a mouse or a keyboard is received (step S310). Even when it is not determined that data has been received from the server device 10 (No in step S303), the operation information receiving unit 21 waits until operation information is received (step S310).

In the present embodiment, the calculation and notification of the data transfer speed are performed after the display of the image data, but the present invention is not limited to this. For example, the calculation and notification of the data transfer rate may be performed at an arbitrary timing as long as the image data is received from the server device 10.

According to the present embodiment, either the still image compression method or the moving image compression method is selected as the compression method for the frequently changed region based on the estimated response time calculated in advance. Specifically, when the estimated response time exceeds the time threshold value, the compression method for still images is selected as the compression method for the high-frequency change area, not the compression method for moving images. Since the still image compression method requires less calculation amount than the moving image compression method, the processing time of the server device 10 is shortened. For this reason, since the response time from when the operation instruction of the client terminal 20 is received until the execution result of the server device 10 is displayed on the client terminal 20 is shortened, the operation response of the client terminal 20 can be improved.

In addition, as a compression method for image data in the frequently changed region, the still image compression method is used only when the estimated response time exceeds the time threshold, not the still image compression method. Is used. For this reason, when the estimated response time does not exceed the time threshold, that is, when the operation response is good, the moving image compression method is used. Therefore, the amount of data transferred from the server device 10 to the client terminal 20 does not increase due to the selection of the still image compression method.

In the above embodiment, the transfer rate calculation unit 235 of the client terminal 20 calculates the data transfer rate based on the packet transmission time and reception time and the data amount of the packet image data. However, the present embodiment is not limited to this. For example, if the data transfer rate is managed by the OS 1100, the client terminal 20 may directly acquire the data transfer rate. *

In the above embodiment, the still image compression method and the moving image compression method are exemplified, but the present invention is not limited to this. Any compression method may be used as long as the compression rate is different.

Furthermore, in this embodiment, the amount of image data is reduced by compressing the image data. However, if the amount of data can be reduced, processing other than compression processing may be used.

In this embodiment, the response time is defined as the time required from the start of compression of the image data in the frequently changed area to the acquisition of the image data by the client terminal 20. However, the response time is particularly limited as long as it constitutes a part of the time from the operation instruction from the client terminal 20 until the execution result of the application program by the server device 10 is displayed on the client terminal 20. It is not a thing. For example, the time from the operation instruction of the client terminal 20 until the client terminal 20 receives the execution result of the application program by the server device 10 may be set as the response time. Further, the response time may be the compression time of the image data in the high-frequency change area. These response times are also examples of response time.

10: Server device (information processing device)
20: Client terminal (terminal device)
156: Image transmission unit (transmission unit)
157: Compression method determination unit (determination unit)

Claims (11)

1. In the information processing apparatus for causing the terminal device to display an execution result based on an operation instruction from the terminal device,
   A determination unit that determines a processing method related to a data amount of the image data of the execution result based on an index related to a response time to an operation instruction from the terminal device;
   A transmission unit that transmits image data processed by the determined processing method to the terminal device;
   An information processing apparatus comprising:
2. In the information processing apparatus according to claim 1,
   The determination unit performs either the first method or the second method in which the amount of data after processing the image data is smaller than that of the first method based on the response time index. Information processing device to select as a method.
3. In the information processing apparatus according to claim 2,
   The determination unit
   Obtaining a processing time for processing the image data by the first method;
   Calculating a transmission time for transmitting the amount of data of the image data after processing according to the first method to the terminal device;
   An information processing apparatus that calculates an index related to the response time based on the processing time and the transmission time.
4. In the information processing apparatus according to claim 2 or 3,
   When the index related to the response time is an index that increases with an increase in the response time,
   The information processing apparatus, wherein the determination unit selects the second method as the processing method when an index related to the response time exceeds a threshold value.
5. The information processing apparatus according to claim 1, further comprising:
   An acquisition unit that acquires a change area between the image data based on the image data of the execution result,
   When the acquisition unit detects a plurality of change areas,
   The determination unit
   Based on the index related to the response time, for each of the change areas, either the first method or the second method having a smaller data amount after processing the image data than the first method, Information processing apparatus to be selected as a processing method.
6. The information processing apparatus according to claim 5,
   The determination unit
   Obtaining a total processing time for processing the image data of the plurality of change areas by the first method;
   Calculating a transmission time for transmitting the total amount of data after processing according to the first method of the image data of the plurality of change areas to the terminal device;
   An information processing apparatus that calculates an index related to the response time based on the total processing time and the transmission time.
7. The information processing apparatus according to claim 6,
   When the index related to the response time is an index that increases with an increase in the response time,
   The information processing apparatus, wherein the determination unit selects the second method as a processing method of at least any one of the plurality of change areas when an index related to the response time exceeds a threshold value.
8. The information processing apparatus according to claim 5,
   The determination unit
   For each change area, obtain a processing time for processing the image data of the change area by the first method,
   The total processing time for processing the image data of the plurality of change areas by the first method is calculated by accumulating the processing time acquired for each change area,
   Calculating a transmission time for transmitting the total amount of data after processing according to the first method of the image data of the plurality of change areas to the terminal device;
   Calculate the total time of the total processing time and the transmission time,
   When the total time exceeds a threshold value, the first processing method for processing image data in the change area excluding the change area for which the processing time was calculated last, out of the change areas for which the processing time has been calculated, As the processing method of the image data of the change area excluding the change area for which the first method is selected from among the plurality of change areas, the processing of the image data is more than the first method. An information processing apparatus that selects a second method with a small amount of data later.
9. In the information processing apparatus according to claim 8,
   The determination unit is an information processing apparatus that acquires the processing time in order from a larger area among the plurality of change regions.
10. In the control method of the information processing apparatus for causing the terminal device to display the execution result based on the operation instruction from the terminal device,
    Based on an index related to a response time to an operation instruction from the terminal device, a processing method related to a data amount of the image data of the execution result is determined,
    A method for controlling an information processing apparatus, which transmits image data processed by the determined processing method to the terminal apparatus.
11. In the control program of the information processing apparatus for causing the terminal device to display the execution result based on the operation instruction from the terminal device,
    The control program is stored in the information processing apparatus.
    Based on an index related to a response time to an operation instruction from the terminal device, the processing method related to the data amount of the image data of the execution result is determined,
    A control program for an information processing apparatus that causes the terminal apparatus to transmit image data processed by the determined processing method.

Images