JP2011198120A - Image transfer device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve efficient image transfer by applying a relatively high image quality to an area which a user may want to attentively view really, and applying a relatively low image quality to the other areas.SOLUTION: An image transfer device includes a communication processing part 105 for transferring a moving image of a GUI (graphical user interface) of an operating system to an image display device and a work area detection part 106 for detecting a work area of a user in the GUI. The moving image to be transferred is generated with a relatively high image quality with respect to the detected work area and with a relatively low image quality with respect to areas other than the work area.

Description

  The present invention relates to an image transfer apparatus, method, and program, and more particularly, to a technique for efficiently transferring a GUI (graphical interface) screen.

  As technologies related to the present invention, there are technologies described in Patent Documents 1 to 3. In Patent Document 1, the screen is divided into blocks, priorities are set for each block according to the number of times the screen has changed, and the screen of the block with the lower number of screen changes is preferentially transmitted. The method is described.

  Patent Document 2 describes that brightness control is performed to reduce the brightness of an area excluding an active window in a GUI employing a multi-window method (Patent Document 2 is expressed as “multi-window display device”). .

  As a screen transmission method, there is a method using a moving image compression method (interframe prediction) such as MPEG (for example, Patent Document 3). In this method, the entire latest frame (screen) is compared with the previous frame (that is, an image obtained by encoding and decoding the previous frame), and the difference is compressed and transmitted. That's it. In this method, when the data amount of the difference between frames is larger than the transmission band, the image quality is lowered by increasing the quantization step at the time of compression, and when the difference is small and the band becomes free, the image quality is increased. The quantization step is reduced. According to this method, the update frequency of the image data to be transmitted does not greatly decrease compared to the actual screen update frequency due to the limitation of the bandwidth that can be used for screen transfer. Image transmission becomes possible.

JP 2004-086550 A Japanese Patent Laid-Open No. 03-282586 JP 2009-224847 A

  Conventionally, in a thin client system that employs a server / client system configuration, the client accesses the server, captures the contents of work on the server to the screen, transfers the captured data to the client, There is a technology to display the screen. This is an image transfer technique for transferring an image on a GUI display screen. In this image transfer technique, it is required to efficiently use the bandwidth of the transmission path between the server and the client. Therefore, it has been considered to compress and transmit at a relatively high frame rate for a part where the movement in the capture screen is frequent and a relatively low frame rate for a part where the movement is not frequent.

  However, with such a method, it is possible to transmit an area where movement is frequent even though the user is not paying much attention with high image quality, resulting in a decrease in efficiency and conversely impairing usability. There was sex. This will be described with reference to FIG. Generally, there is a clock gadget as one of so-called gadget applications on the GUI desktop (upper right in FIG. 5). The clock gadget is an application that displays an analog clock on the desktop. However, since the second hand moves every second, it is transmitted with high image quality, and the bandwidth of the transmission path cannot be used efficiently.

  Since window-x in FIG. 5 is a so-called active window, it is almost certainly a window that the user is working on. On the other hand, window-y and z happen to be inactive windows, but the user may be working while comparing with window-x. Therefore, it is necessary to appropriately detect a region that the user originally wants to pay attention to.

  The present invention has been made in view of the above-described circumstances, and applies a relatively high image quality to an area that a user originally wants to pay attention to, and is relative to other areas. An object of the present invention is to provide an image transfer apparatus, method, and program for realizing efficient image transfer by applying low image quality to the image.

  In order to achieve the above object, the present invention provides, as a first aspect, a transfer means for transferring a moving image of a GUI (Graphical User Interface) of an operating system to an image display device, and a user work area in the GUI. A work area detecting means for detecting, and generating a moving image to be transferred by the transfer means with a relatively high image quality for the detected work area and a relatively low image quality for an area excluding the work area. And a moving image generation means. An image transfer apparatus is provided.

  In addition, as a second aspect, the present invention provides a work area detection step for detecting a user work area in a GUI (graphical user interface) of the operating system, and is relatively high with respect to the detected work area. A moving image generating step of generating a moving image of the GUI with a low image quality relative to an area excluding the work area, and a transferring step of transferring the generated moving image to an image display device. It is an object of the present invention to provide an image transfer method.

  According to a third aspect of the present invention, as a third aspect, a computer detects a work area detection process for detecting a user work area in a GUI (graphical user interface) of an operating system, and a relative to the detected work area. Moving image generation processing for generating the GUI moving image at a relatively high image quality and relatively low image quality relative to the area excluding the work area, and transfer processing for transferring the generated moving image to the image display device And providing a program characterized in that the program is executed.

  According to the present invention, a relatively high image quality is applied to an area that the user originally wants to pay attention to, and a relatively low image quality is applied to other areas so as to be efficient. It is possible to provide an image transfer apparatus, method, and program for realizing image transfer.

It is a block diagram (the 1) which shows the structure of one Embodiment which implemented this invention suitably. It is a block diagram (the 2) which shows the structure of one Embodiment which implemented this invention suitably. It is a figure which shows an example of the window management table which the work area detection part 106 of this embodiment has. FIG. 4 is a diagram showing a flow of window change detection processing executed by a work area detection unit 106 to generate the window management table of FIG. 3. It is a figure for pointing out the problem of a prior art, and is a figure which shows the whole area | region of a desktop screen.

  1 and 2 show the configuration of an image transfer / display system 1 according to an embodiment in which the present invention is suitably implemented. FIG. 1 is a diagram illustrating a configuration in which different compression rates are applied to an area that is considered to be a user's work and an area that is not. On the other hand, FIG. 2 is a diagram illustrating a configuration in which encoding is performed at different frame rates for an area that is considered to be a user's work and an area that is not.

  The image transfer / display system 1 is an embodiment in which the present invention is applied to, for example, a thin client system adopting a server / client system configuration. In the present embodiment, the client accesses the server, captures the contents of work on the server on the screen, transfers the captured data to the client, and displays the server screen on the client. The image transfer apparatus 100 in the figure corresponds to the server, and the image display apparatus 200 corresponds to the client.

The image transfer apparatus 100 includes a screen capture unit 101, a previous screen data storage unit 102, an update data generation processing unit 103, an image compression processing unit 104, a communication processing unit 105, and a work area detection unit 106.
The screen capture unit 101 captures a screen displayed on the image display apparatus 200 side.
The previous screen data storage unit 102 takes a difference from the previous screen data and stores it in order to efficiently compress the data amount of the moving image.

  The update data generation processing unit 103 compares the previously captured image data with the currently captured image data and extracts the difference at a certain point in time. This difference is called update data, and the process of extracting the difference is called update data generation process. In the present embodiment, the update data generation processing unit 103 performs update data generation processing at a frame rate specified by the work area detection unit 106. In addition, when the work area detection unit 106 divides the entire screen into a plurality of areas and designates update data generation processing to be performed at different timing for each area, the update data generation processing unit 103 updates at a different frame rate for each area. Perform data generation processing.

  The image compression processing unit 104 uses screen image data (key frames) captured by the screen capture unit 101 and update data generated by the update data generation processing unit 103 to transfer data to the image display device 200. Compress the amount. In the present embodiment, the image compression processing unit 104 performs compression processing using the compression parameters specified by the work area detection unit 106. In addition, when the work area detection unit 106 divides the entire screen into a plurality of areas and specifies to compress the data amount at different compression rates for each area, the image compression processing unit 104 performs compression processing with different compression parameters for each area. I do.

  The communication processing unit 105 performs communication processing for transferring the moving image data compressed by the image compression processing unit 104 to the image display apparatus 200 by packetizing the data as transfer data. Various communication paths such as wired or wireless can be used. A so-called home LAN may be used.

  The work area detection unit 106 performs a process for detecting an area in which the user is working (referred to as “work area detection process”), and divides the entire screen into a plurality of areas based on the detection result. The update data generation process is specified at different timings, or the process of specifying the data amount to be compressed at a different compression rate for each region is performed.

The image display apparatus 200 includes a communication processing unit 201, an image decoding processing unit 202, an image composition processing unit 203, a previous screen data storage unit 204, and an image display unit 205.
The communication processing unit 201 performs processing such as receiving transfer data from the image transfer apparatus 100, extracting moving image data from the packet, and passing it to the decoding process.
The image decoding processing unit 202 performs a process of decoding and expanding the compressed moving image data. Since the decoded data uses the difference information, the image decoding processing unit 202 once stores the previous screen data in the previous screen data storage unit 204. Further, the difference information is input to the image composition processing unit 203.

The image composition processing unit 203 generates an image to be displayed by combining the previous screen data stored in the previous screen data storage unit 204 and the difference information input from the image decoding processing unit 202. Process.
The image display unit 205 displays the image synthesized by the image synthesis processing unit 203.

The main points will be summarized again with reference to FIGS.
In the configuration shown in FIG. 1, the compression parameter is compressed as the first compression rate for the user work area based on the user work area extracted by the work area detection unit 106, while the user work area is outside the user work area. Compresses the compression parameter as the second compression rate. Here, it is assumed that the first compression rate is lower than the second compression rate.

In the configuration shown in FIG. 2, update data generation processing is performed at the first timing for the user work area based on the user work area extracted by the work area detection unit 106, while the outside of the user work area is performed. Update data generation processing is performed at the second timing. Here, the first timing is assumed to be more frequent than the second timing.
Specifically, for the user work area, the difference detection process (update data generation process) is performed for all frames, and for the outside of the user work area, the difference detection process is performed with respect to the number of frames. Good. For example, when the number of frames per unit time is 30 fps, the difference detection process (update data generation process) is performed only once in 10 frames outside the user's work area. In this case, the difference detection process (update data generation process) is performed outside the user's work area at a cycle of 3 fps.
As described above, the update data generation processing unit 103 and the image compression processing unit 104 function as a moving image generation unit.

Next, the process (work area detection process) of the work area detection unit 106 will be described. “Working area” refers to an area where the user is considered working.
The work area detection unit 106 has a window management table shown in FIG. FIG. 3 is an example of a window management table. The window management table in FIG. 3 includes a window identifier, start point coordinates, and end point coordinates. In addition, the arrangement order of each record (each element) is the arrangement order of the second window from the last window, the third window from the last window, etc. That is, the window at the top is the so-called “active window”. The second and subsequent rows are “history of windows that were active windows”.

The window identifier is an identifier for identifying each window. For example, a window handle or the like is used in Windows (registered trademark) of Microsoft Corporation. The work area detection unit 106 uses this window identifier to detect which window is changing when the window is changed.
The start point coordinates are the upper left coordinates of the window, and the end point coordinates are the lower right coordinates. That is, it is only necessary to specify the position or range of the window in the entire displayable area (the entire screen).

  The work area detection unit 106 detects a work area using the window management table. The work area detection unit 106 determines that an area obtained by adding the areas indicated by the start point coordinates and the end point coordinates of each window registered in the window management table is the work area. That is, the work area is a union of areas that each window occupies in the entire screen. That is, a collection of points that belong to any area of the window registered in the window management table is set as a work area. As another embodiment, the work area may be determined from the leftmost coordinate, the rightmost coordinate, the uppermost coordinate, and the lowest coordinate of each window. When the work area is determined by this method in the example shown in FIG. 3, a rectangular area having a start point coordinate (0, 0) and an end point coordinate (520, 789) is obtained.

  In some cases, the window management table is empty and there is no window. In this case, the entire screen is determined as the work area. In the subsequent update data generation processing unit 103 and the image compression processing unit 104, the same image quality is applied to the entire screen, and the screen is uniformly transmitted to the image display apparatus 200.

FIG. 4 shows a flow of window change detection processing executed by the work area detection unit 106 to generate the window management table. The main subject of the processing of each processing block shown in FIG.
The window change detection process shown in FIG. 4 starts when a window change is detected using, for example, a window handle. The work area detection unit 106 detects a window change from the operation system (step S101).

  When the change of the active window is detected (step S102 / Yes), it is searched whether there is a changed active window in the window management table (step S103). This search is performed by searching for a window having the same window identifier.

  When the same window exists in the window management table (step S104 / Yes), the window is moved to the head of the window management table (step S105). If the same window does not exist in the window management table (step S104 / No), the newly detected window is added to the head of the window management table (step S106). At this time, the window identifier and start point / end point coordinate information are also added. Further, when the window management table area becomes insufficient due to the addition, the information of the last window is deleted.

  If the window coordinates are changed due to a window size change or the like (Yes in step S107), the window management table is searched for a window whose coordinates have changed (step S108). This search is performed by searching for a window having the same window identifier.

  If the same window exists in the window management table (step S109 / Yes), the coordinates of the window are changed (step S110). Coordinates are changed by overwriting the window management table with the changed start point coordinates and end point coordinates. If the same window does not exist in the window management table (step S109 / No), no processing is performed.

  When the closing of the window is detected (step S111 / present), or when the minimization of the window is detected (step S112 / present), the window management table is searched for a window for which the closing or minimizing is detected (step S112 / present). Step S113). This search is performed by searching for a window having the same window identifier.

  If there is a closed window (step S114 / Yes), information about the window is deleted from the window management table (step S115). If the same window does not exist in the window management table (step S114 / No), no processing is performed.

  When there is no active window (step S116 / Yes), all the information in the window management table is deleted (step S117).

This is the end of the description of the work area detection processing performed by the work area detection unit 106.
Note that the work area may be adjusted to an appropriate work area according to the user's required level of image quality, the state of the bandwidth of the transmission path, and the like. Adjustment is performed by the number of records (elements) added to the window management table. When the number of records (elements) added to the window management table is 1, only the active window is always regarded as a work area, and image transfer is performed by applying high image quality to the active window and applying low image quality to the other areas. When the number is 2 or more (however, a finite number), in addition to the active window, only relatively new ones in the history of windows of applications that the user has worked in the past are regarded as work areas. If the number is infinite, all windows of applications that the user has worked in the past (windows that have become active in the past) are stored, and the overlapped area is regarded as a work area.

  According to the above-described embodiment, a relatively high image quality is applied to an area that the user originally wants to watch, that is, a work area, and a relatively low image quality is applied to other areas. It can be applied to achieve efficient image transfer.

  Note that the image transfer apparatus 100 and the image display apparatus 200 can also be realized by a program that causes a computer to execute each process described in the above embodiment using a general-purpose computer.

DESCRIPTION OF SYMBOLS 1 Image transfer / display system 100 Image transfer apparatus 101 Screen capture part 102 Previous screen data storage part 103 Update data generation process part 104 Image compression process part 105 Communication process part 106 Work area detection part 200 Image display apparatus 201 Communication process part 202 Image Decoding processing unit 203 Image composition processing unit 204 Previous screen data storage unit 205 Image display unit

Claims (8)

  Detects the user's work area in the GUI (graphical user interface) of the operating system and generates a moving image of the GUI after performing high-quality image processing on the work area and low-quality image processing outside the work area. And transferring the moving image to an image display device. Transfer means for transferring a moving image of a GUI (graphical user interface) of the operating system to the image display device;
Work area detection means for detecting a user work area in the GUI;
A moving image generating means for generating a moving image to be transferred by the transfer means with a relatively high image quality for the detected work area and a relatively low image quality for an area excluding the work area;
An image transfer apparatus comprising: The moving image generating means generates a moving image using inter-frame prediction,
For the work area, generate a difference used for the inter-frame prediction at a first timing,
For the area excluding the work area, a difference used for the inter-frame prediction is generated at a second timing,
The image transfer apparatus according to claim 2, wherein the first timing has a higher frequency than the second timing. The moving image generating means
For the work area, compress at a first compression rate,
Compress the area excluding the work area at the second compression rate,
4. The image transfer apparatus according to claim 2, wherein the first compression rate is lower than the second compression rate.   The work area detection means has a window management table for sequentially registering active windows and windows that have become active windows, and the registered window area is the work area. Item 5. The image transfer device according to any one of Items 2 to 4. The window management table has information for specifying the range of each registered window,
6. The image transfer apparatus according to claim 5, wherein the work area detection means sets the area where the ranges of the windows overlap as the work area based on information specifying the window range. A work area detection step for detecting a user work area in a GUI (graphical user interface) of the operating system;
A moving image generation step of generating a moving image of the GUI with a relatively high image quality for the detected work area and a relatively low image quality for an area excluding the work area;
A transfer step of transferring the generated moving image to an image display device;
An image transfer method comprising: On the computer,
A work area detection process for detecting a user work area in a GUI (graphical user interface) of the operating system;
A moving image generation process for generating a moving image of the GUI with a relatively high image quality for the detected work area and a relatively low image quality for an area excluding the work area;
A transfer process for transferring the generated moving image to an image display device;
A program characterized by having executed.

Images