JP2011139361A - Image processor, image processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a transmitting-side processing load and to reduce a receiving-side processing load in a screen sharing system. <P>SOLUTION: Screen data are drawn from video data reproduced by using a drawing command issued by an application, and the drawing command used for drawing is stored. Furthermore, the stored drawing command is analyzed, the screen data are separated into a plurality of regions and an attribute in each of the separated regions is determined. The separated region is then encoded in accordance with the determined attribute and screen data relating to the encoded region are transmitted to a client terminal. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to a technique suitable for use in transmitting screen data from a server to a client terminal.

  Conventionally, many methods for capturing and transferring a screen of a screen sharing server have been proposed. For example, in the thin client system described in Patent Document 1, a video reproduction processing unit is provided in the thin client server device, and a video drawing processing unit is provided in the thin client terminal. Then, on the thin client server device side, a transparent color is set in the video display area, and it is transmitted as screen data via the remote controller, and the video data is transmitted via another route. Thereby, on the thin client terminal side, these screen data and video data are displayed on the same screen by overlay display.

  Further, in the image transmission system described in Patent Document 2, the drawing area where the image has changed is identified as a plurality of rectangular ranges, and the plurality of identified rectangular ranges and the plurality of rectangular ranges read from the drawing range set storage unit are identified. Compare with set. Furthermore, a new set of rectangular ranges is reconstructed and stored. When it is notified that transmission is possible, the reconstructed set of rectangular ranges is read, and images in a plurality of rectangular ranges are read and the images are compressed.

JP 2007-31957 A JP 2005-339256 A

  Conventionally, changes in drawn image data (bitmap data or RAW data) are superimposed in a rectangular area in the time direction and stored in a memory. Then, at the time of transmission, the image data corresponding to the final change is read out in a plurality of rectangular areas, and the plurality of image data is further compressed and transmitted. In this way, the load on the communication path is reduced by transmitting only the image data corresponding to the final change. However, on the receiving side, processing to expand compressed data, processing to synchronize the position where a plurality of rectangular areas are arranged, and the plurality of rectangular areas and image data must be performed, resulting in an enormous processing load. .

  In the configuration described in Patent Document 1, when the position of the video data moves, the position of the screen data to be overlaid also changes. For this reason, there is a problem in that display control processing for synchronization is complicated when video data and screen data sent separately are combined and displayed on the client side. Further, in the configuration described in Patent Document 2, a drawing area where an image has changed is monitored and stored in a rectangular range. For this reason, a large-capacity memory is required to provide a control unit for performing the processing or to store an image in a rectangular range as it is for transmission. As a result, there is a problem that the processing configuration becomes complicated and the cost increases.

  In view of the above-described problems, the present invention has an object to reduce the processing load on the transmission side in the screen sharing system and to reduce the processing load on the reception side.

  An image processing apparatus of the present invention is an image processing apparatus that is connected to a client terminal that receives screen data via a network, reproduces video data, and generates the screen data. A drawing unit that draws screen data using a drawing command issued by the computer, a drawing command storage unit that stores the drawing command used in the drawing unit, and a screen data storage that stores the screen data drawn by the drawing unit Means for analyzing the drawing command stored in the drawing command storage means, and separating the screen data stored in the screen data storage means into a plurality of areas, and stored in the drawing command storage means. Analyzing the drawn command and determining the attribute of each area separated by the area separating means Attribute determining means, encoding means for performing encoding according to the attribute determined by the attribute determining means for the area separated by the area separating means, and the area encoded by the encoding means Transmission means for transmitting the screen data according to the above to the client terminal.

  According to the present invention, it is possible to perform region separation without performing detailed analysis processing on the screen data itself, and to transmit the screen data to the client terminal.

It is a block diagram which shows the structural example of the screen sharing system of 1st Embodiment. It is a figure which shows the operation | movement sequence in a screen sharing system. It is a flowchart which shows an example of the operation | movement procedure which transfers a screen. It is a flowchart which shows an example of the detailed procedure of a screen transfer pre-processing. It is a flowchart which shows an example of the detailed procedure of a screen transfer this process. It is a block diagram which shows the structural example of the screen sharing system of 2nd Embodiment. It is a flowchart which shows an example of the detailed procedure of a screen transfer pre-processing. It is a flowchart which shows an example of the detailed procedure of a screen transfer this process. It is a block diagram which shows the structural example of the screen sharing system of 3rd Embodiment.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration example of a screen sharing system according to the present embodiment.
1, in the screen sharing system according to the present embodiment, a screen sharing server device 1 and a thin client terminal 2 in charge of receiving a shared screen and inputting an operation are connected by a network 3. The screen sharing server device 1 is connected to a display device 4 that performs screen display.

  The screen sharing server device 1 is composed of the following components. The drawing framework 110 is used for drawing various characters, still images, moving images, and the like by using a drawing command and the like and storing them in the frame buffer 140 as frame image data. Here, the drawing command refers to, for example, a display lower layer of a graphic command that directly handles image data that stores a graphic object or a moving image stream of a natural image in the frame buffer 140 as bitmap data (or display raw data). . Note that the drawing command is not limited to the above example. The drawing command storage unit 120 is for temporarily storing the drawing commands executed by the drawing framework 110 while monitoring them. The drawing command storage unit 120 stores the drawing command until the area drawn by the drawing command is erased by the erase command of the same command.

  The capture control unit 130 is for performing settings for copying and transferring the frame image data stored in the frame buffer 140 at a predetermined capture rate. The capture control unit 130 also performs control such as acquiring drawing commands stored in the drawing command storage unit 120. The frame buffer 140 is a memory for storing screen data (frame image data) drawn by a drawing command, and the frame image data is read from the frame buffer 140 when the display device 4 displays a screen. . The video reproduction unit 150 is for reproducing the video data stored in the video data unit 160.

  The transfer buffer 170 functions as screen data storage means, and is a buffer for storing frame image data copied for screen transfer. The remote screen control unit 180 is for encoding the frame image data stored in the transfer buffer 170 for transfer. The remote screen control unit 180 receives various information from the thin client terminal 2 and controls the screen sharing process. The communication unit 190 mainly controls communication with an external device (particularly, the thin client terminal 2) through the network 3 or the like.

  The frame buffer 140 of the screen sharing server apparatus 1 in this embodiment uses a part of main memory such as a RAM (not shown) and is used as a frame buffer in a fixed area.

  FIG. 2 is a diagram showing an operation sequence in the screen sharing system of the present embodiment. FIG. 3 is a flowchart illustrating an example of an operation procedure for transferring a screen by the screen sharing server apparatus 1 according to the present embodiment. Hereinafter, the processing of the screen sharing system according to the present embodiment will be described in detail with reference to FIGS. 2 and 3.

  First, when a login operation is executed by the user in the thin client terminal 2, the remote screen control unit 180 waits until the login request M001 shown in FIG. 2 is received from the thin client terminal 2 via the communication unit 190. (Step S301). When receiving the login request M001, the remote screen control unit 180 determines whether or not to accept the login request from the thin client terminal 2 (step S302). Specifically, the ID and password received as parameters of the login request M001 are authenticated to determine whether they are valid. As a result of this determination, if the ID and password are not valid and the login request is not accepted, the process ends.

  On the other hand, if the ID and password received as parameters of the login request M001 are valid as a result of the determination in step S302 and the login request is accepted, screen transfer pre-processing is performed (step S303). This screen transfer pre-processing will be described later in the description of FIG. Next, screen transfer main processing is performed (step S304). This screen transfer main processing will also be described later in the description of FIG. Through this screen transfer process, the thin client terminal 2 receives the screen data encoded from the screen sharing server device 1 via the network 3, decodes the received screen data, and displays the screen.

  When a logout operation is executed by the user in the thin client terminal 2, a remote control event (logout) M004 shown in FIG. 2 is transmitted to the screen sharing server apparatus 1 via the network 3. Therefore, the remote screen control unit 180 determines whether or not the remote control event (logout) M004 is received from the thin client terminal 2 via the communication unit 190 (step S305). If the result of this determination is that a remote control event (logout) M004 has not been received, processing returns to step S304.

  On the other hand, if the result of determination in step S305 is that a remote control event (logout) M004 has been received, the remote screen control unit 180 performs screen transfer processing end processing (step S306) and ends processing. In step S306, the following processing is specifically executed.

  First, the remote screen control unit 180 notifies the capture control unit 130 of the end of capture. Then, the capture control unit 130 instructs the drawing framework 110 to end duplication of the frame image data stored in the frame buffer 140. Furthermore, the capture control unit 130 instructs the drawing command storage unit 120 to delete the stored drawing command and stop monitoring and temporarily storing the drawing command.

FIG. 4 is a flowchart showing an example of a detailed procedure of the screen transfer pre-processing in step S303 of FIG.
First, in FIG. 4, the remote screen control unit 180 acquires current resource information in the screen sharing server apparatus 1 (step S401). Then, the screen transfer parameter is determined based on the device information of the thin client terminal 2 received as the parameter of the login request M001 and the current resource information of the screen sharing server device 1. Specifically, the angle of view, the capture rate, etc. of the image for screen sharing that are optimal for the screen sharing system such as the specifications of the thin client terminal 2 are determined (step S402).

  Next, the remote screen control unit 180 notifies the capture control unit 130 of capture information such as an angle of view and a capture rate of an image for screen sharing. Further, screen transfer parameter information is generated from the capture information to the thin client terminal 2 and transmitted through the communication unit 190 together with the login confirmation M002 shown in FIG. 2 (step S403). At this time, the thin client terminal 2 receives the login confirmation M002 and the screen transfer parameter information from the screen sharing server device 1 via the network 3. Various settings are made based on the screen transfer parameter information, and the screen to be transferred is awaited.

  Next, the capture control unit 130 instructs the drawing framework 110 to start duplicating the frame image data stored in the frame buffer 140 at the capture rate determined in step S402 (step S404). With the above processing, the screen transfer pre-processing is completed.

FIG. 5 is a flowchart showing an example of a detailed procedure of the screen transfer main process in step S304 of FIG.
First, the capture control unit 130 instructs the drawing command storage unit 120 to temporarily store drawing commands executed in the drawing framework 110 while monitoring them (step S501). The timing at which a drawing command is deleted from the drawing command storage unit 120 is assumed when an end command is executed among the monitored drawing commands, or when an application that has issued the drawing command is ended.

  When the thin client terminal 2 receives the login confirmation M002, the thin client terminal 2 transmits a remote control event M003. By this remote control event M003, the application software operates in the screen sharing server device 1 under the control of the remote screen control unit 180, and the reproduction of the video data stored in the video data unit 160 of the screen sharing server device 1 is instructed. The In this case, the video playback unit 150 reads designated video data from the video data unit 160, and executes a drawing command related to video playback by the drawing framework 110.

  Thereby, the capture control unit 130 detects the operation of the application and determines whether or not a command related to region separation such as a drawing command related to video reproduction is stored in the drawing command storage unit 120 (step S502).

  As a result of this determination, when a command related to area separation such as a drawing command related to video reproduction is not stored in the drawing command storage unit 120, normal transfer processing is performed. In the normal transfer process, the remote screen control unit 180 performs a single encoding process on the frame image data copied to the transfer buffer 170 (step S503). And it transmits to the thin client terminal 2 via the communication part 190 and the network 3 (step S506).

  On the other hand, as a result of the determination in step S502, when a command related to area separation such as a drawing command related to video reproduction is stored in the drawing command storage unit 120, the following processing is executed. First, when the frame image data is duplicated in the transfer buffer 170, the remote screen control unit 180 adjusts the duplicated frame image to the angle of view for transfer determined in step S402, and so on. Is used to perform image processing such as resizing. In addition, a command related to region separation such as a drawing command related to video reproduction temporarily stored in the drawing command storage unit 120 is analyzed in detail. Then, separation processing and attribute determination of areas such as a character attribute portion and a natural image attribute portion are performed on the frame image data stored in the transfer buffer 170 based on the analysis result (step S504). Next, an appropriate encoding method is applied to the attribute of each area, and the encoding of each area is executed by the encoder of the remote screen control unit 180 (step S505). Then, the image data of the target area is transmitted to the thin client terminal 2 via the communication unit 190 and the network 3 (step S506).

  Here, in the region separation processing described above, the overlapping and overlapping order of regions of drawing commands are also considered as parameters for separating the regions. In this way, unlike the method of capturing only the object to be drawn when issuing the drawing command, it is easy to obtain the position / overlapping relationship between the areas and the frame synchronization by temporarily storing them in the frame buffer. 2 is simplified.

  The area of the natural image attribute portion is encoded by an intra-frame compression method so that the natural image is clearly displayed. For example, by using DCT (discrete cosine transform) encoding, Huffman encoding (arithmetic compression encoding), JPEG compression combining these two, wavelet transform encoding, and JPEG compression parameters optimal for natural images, A natural picture can be displayed more beautifully. On the other hand, the area of the character attribute part is encoded by a compression method so that the characters are clearly displayed. For example, by using a lossless compression method such as dictionary compression such as LZ or LZW or run-length compression, the character area can be displayed more clearly.

  In the present embodiment, regions such as a character attribute portion and a natural image attribute portion are separated by a rectangular region. On the other hand, when the area drawn by the drawing command is other than a rectangle, for example, the area separation is performed by approximating the outline of the drawn drawing area that is not a rectangle.

  As described above, according to the present embodiment, when capturing from the drawing framework 110, replication is performed using a drawing command. As a result, it is not necessary to take time to lock overwriting of the frame buffer for the time to copy, rather than to copy the image data stored in the frame buffer as it is. For this reason, processing delay can be reduced.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 6 is a block diagram illustrating a configuration example of the screen sharing server device 6 in the present embodiment. It is assumed that the screen sharing server device 1 of the present embodiment is a blade server type that enters a server rack or the like. The difference from the screen sharing server apparatus 1 according to the first embodiment is that the interface connected to the display apparatus 4 and the video data unit 160 are not provided. Further, in place of the transfer buffer 170 of FIG. 1, a network video processing unit 175 for performing graphic and video decoding processing for the network is provided. The network video processing unit 175 is equipped with a graphic RAM, and a mounting form includes a dedicated PCI-Express card, a graphic processor chip, a main general-purpose MPU, a multi-core mounted graphic core, and the like. Further, in place of the remote screen control unit 180 of FIG. 1, a remote terminal control unit 185 that performs various management including the device of the thin client terminal 2 on the remote operation side is provided. Since the configuration other than this is the same as that of the screen sharing server apparatus 1 of FIG. 1, detailed description thereof is omitted.

  The operation sequence of the screen sharing server device 6 according to the present embodiment is the same as that shown in FIG. Furthermore, the processing procedure from login to logout by the screen sharing server device 6 according to the present embodiment is also the same as that in FIG. In the process of FIG. 3, the process performed by the remote screen control unit 180 is performed by the remote terminal control unit 185 in this embodiment.

  FIG. 7 is a flowchart illustrating an example of a detailed procedure of the screen transfer pre-processing in step S303 of FIG. 3 in the present embodiment. Steps S701 to S703 are the same as steps S401 to S403 in FIG.

  Then, the capture control unit 130 instructs the drawing framework 110 to start writing the frame image data to the network video processing unit 175 at the capture rate and the angle of view determined in step S702 (step S704). With the above processing, the screen transfer pre-processing is completed.

FIG. 8 is a flowchart illustrating an example of a detailed procedure of the screen transfer main process in step S304 of FIG. 3 in the present embodiment.
First, the capture control unit 130 instructs the drawing command storage unit 120 to temporarily store a drawing command to be executed to the network video processing unit 175 in the drawing framework 110 (step S801). The drawing command is the same as in the first embodiment, and the timing at which the drawing command is deleted from the drawing command storage unit 120 is also the same.

  When the thin client terminal 2 receives the login confirmation M002, the thin client terminal 2 transmits a remote control event M003. By this remote control event M003, the application software operates in the screen sharing server apparatus 1. Then, the remote terminal control unit 185 instructs the video reproduction unit 150 to acquire and reproduce video data from the outside via, for example, a network. In this case, the video playback unit 150 acquires designated video data from the outside via a network or the like, and the drawing framework 110 executes a drawing command related to video playback.

  Thereby, the capture control unit 130 detects the operation of the application, and determines whether or not a command related to region separation such as a drawing command related to video reproduction is stored in the drawing command storage unit 120 (step S802).

  As a result of this determination, when a command related to area separation such as a drawing command related to video reproduction is not stored in the drawing command storage unit 120, normal transfer processing is performed. In the normal transfer process, the frame image data drawn on the network video processing unit 175 is subjected to image encoding preprocessing such as adjustment of encoding parameters in accordance with the status of the bandwidth of the network 3 by the remote terminal control unit 185 ( Step S803). The frame image data drawn on the network video processing unit 175 is subjected to a single encoding process using the adjusted encoding parameter, and the thin client terminal 2 is connected via the communication unit 190 and the network 3. (Step S804).

  On the other hand, as a result of the determination in step S802, if a command related to area separation such as a drawing command related to video reproduction is stored in the drawing command storage unit 120, the following processing is executed. First, when frame image data is drawn on the network video processing unit 175, the remote terminal control unit 185 adjusts the frame image data before image encoding such as adjustment of encoding parameters in accordance with the status of the bandwidth of the network 3, etc. Processing is performed (step S805).

  In addition, a command related to region separation such as a drawing command related to video reproduction temporarily stored in the drawing command storage unit 120 is analyzed in detail. Then, separation processing and attribute determination in a plurality of regions such as a character attribute part and a natural image attribute part are performed on the frame image data drawn on the network video processing unit 175 based on the analysis result (step S806). Next, an appropriate encoding method is applied to the attribute of each area, and the encoding process of each area is performed using the encoding parameter adjusted in step S805 (step S807). Then, the remote terminal control unit 185 transmits the image data of the target area to the thin client terminal 2 via the communication unit 190 and the network 3 (step S808). Note that the region separation processing and encoding processing are the same as those in the first embodiment.

  As described above, according to the present embodiment, image data can be drawn only for a thin client terminal without connecting a display on the screen sharing server device side. Thereby, it becomes easier and faster, and it is possible to provide time for performing various processes specialized for the network.

(Third embodiment)
FIG. 9 is a block diagram illustrating a configuration example of the screen sharing system according to the present embodiment.
The difference from the first embodiment is that the screen sharing server device 9 of the present embodiment is not equipped with a function of copying a frame image in the drawing framework 110, but is provided with a capture unit 200 instead. As described above, since the drawing framework 110 does not have a function of copying the frame image data, the video processing unit 210 reads the frame image from the frame buffer 140. At this timing, the capture unit 200 copies the frame image data to the transfer buffer 170. Thereby, the same processing as in the first embodiment can be performed.

  Further, as shown in FIG. 9, the transfer buffer 170, the remote screen control unit 180, and the communication unit 190 in the first embodiment are mounted as modules (hardware modules or cards) outside the screen sharing server device 9. ing. Even in such a case, the same processing as in the first embodiment can be performed. The same applies to the configuration of the second embodiment.

(Other embodiments)
In the above-described embodiment, as an encoding method suitable for each divided region, a method specialized in the attribute of each region is used as the encoding method itself, from the one in which the JPEG parameter is changed. Also good. Examples of the encoding method itself include JPEG, LZH, ZLIB, JPEG2000, PNG, GIF, WMF, AVC Intra (H.264 / AVC Intra), and the like.

  In the above-described embodiment, the drawing command may be a GDI command or a DirectX or DirectShow level command in Microsoft Windows (registered trademark). Also, OpenMAX-il, etc. developed by Khronos Group, or OpenGL commands may be used.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

110 drawing framework, 120 drawing command storage unit, 140 frame buffer, 180 remote screen control unit

Claims (10)

An image processing apparatus that is connected to a client terminal that receives screen data via a network, reproduces video data, and generates the screen data.
Drawing means for drawing screen data from the reproduced video data using a drawing command issued by an application;
Drawing command storage means for storing drawing commands used in the drawing means;
Screen data storage means for storing screen data drawn by the drawing means;
An area separating unit that analyzes the drawing command stored in the drawing command storage unit and separates the screen data stored in the screen data storage unit into a plurality of regions;
An attribute determination unit that analyzes the drawing command stored in the drawing command storage unit and determines an attribute of each region separated by the region separation unit;
Encoding means for performing encoding according to the attribute determined by the attribute determination means for the area separated by the area separation means;
An image processing apparatus comprising: a transmission unit configured to transmit screen data relating to an area encoded by the encoding unit to the client terminal. Receiving means for receiving device information from the client terminal;
Determining means for determining screen parameters according to the specifications of the client terminal based on the device information of the client terminal received by the receiving means;
The image processing apparatus according to claim 1, wherein the drawing unit draws screen data according to the parameter determined by the determination unit.   The image processing apparatus according to claim 1, wherein the area separation unit separates the screen data stored in the screen data storage unit into rectangular areas.   The region separating means, when analyzing the drawing command, if the region to be separated is not a rectangular region, the region separating unit separates the region by a rectangle so as to surround an outer shape of the region that is not the rectangular region. The image processing apparatus according to 3.   The image processing apparatus according to claim 1, wherein the encoding unit performs encoding by lossless compression when the attribute is a character attribute.   The image processing apparatus according to claim 1, wherein the encoding unit performs encoding by intra-frame compression when the attribute is a natural image attribute.   The image processing apparatus according to claim 1, wherein the drawing command is a graphic command.   The image processing apparatus according to claim 1, wherein the drawing unit starts drawing in response to receiving a remote operation from the client terminal. An image processing method in an image processing apparatus that is connected to a client terminal that receives screen data via a network and reproduces video data to generate the screen data,
A drawing step of drawing screen data from the reproduced video data using a drawing command issued by an application;
A drawing command storage step of storing the drawing command used in the drawing step in a storage means;
A screen data storage step of storing the screen data drawn in the drawing step in a storage means;
A region separation step of analyzing the drawing command stored in the drawing command storage step and separating the screen data stored in the screen data storage step into a plurality of regions;
An attribute determination step of analyzing the drawing command stored in the drawing command storage step and determining an attribute of each region separated in the region separation step;
An encoding step for performing encoding according to the attribute determined in the attribute determination step for the region separated in the region separation step;
An image processing method comprising: a transmission step of transmitting screen data related to the region encoded in the encoding step to the client terminal. A program for controlling an image processing apparatus that is connected to a client terminal that receives screen data via a network, reproduces video data, and generates the screen data.
A drawing step of drawing screen data from the reproduced video data using a drawing command issued by an application;
A drawing command storage step of storing the drawing command used in the drawing step in a storage means;
A screen data storage step of storing the screen data drawn in the drawing step in a storage means;
A region separation step of analyzing the drawing command stored in the drawing command storage step and separating the screen data stored in the screen data storage step into a plurality of regions;
An attribute determination step of analyzing the drawing command stored in the drawing command storage step and determining an attribute of each region separated in the region separation step;
An encoding step for performing encoding according to the attribute determined in the attribute determination step for the region separated in the region separation step;
A program causing a computer to execute a transmission step of transmitting screen data relating to an area encoded in the encoding step to the client terminal.

Images