WO2010050593A1

WO2010050593A1 - Communication apparatus and display apparatus

Info

Publication number: WO2010050593A1
Application number: PCT/JP2009/068699
Authority: WO
Inventors: 泰如西林; 村井　信哉; 後藤　真孝; 山口　健作; 博史川添
Original assignee: 株式会社東芝
Priority date: 2008-10-30
Filing date: 2009-10-30
Publication date: 2010-05-06
Also published as: JP5159562B2; JP2010109637A

Abstract

Provided is a communication apparatus comprising: an updated image generation unit (111a) for generating an updated image in order to update a displayed image; a difference region detection unit (112) for detecting a difference region between an updated image and a displayed image; a processing time calculation unit (115) for calculating the software processing time when an image of a plurality of difference regions is expanded using software, and the hardware processing time when an image of an integrated region that includes the plurality of difference regions is expanded using hardware in the case where the plurality of difference regions are detected; a region integration determination unit (114) for determining whether the hardware processing time is shorter than the software processing time; a compressed image generation unit (113) for generating a compressed image in which the image of the integration region is compressed when the hardware processing time is shorter than the software processing time; and a transmission unit (117a) for transmitting the compressed image to a display apparatus.

Description

Communication device and display device

The present invention relates to a communication device and a display device that realize a function of sharing an application screen between devices.

For the purpose of improving usability, there is a computing system in which a terminal device having a minimum input / output interface is arranged on the user side, and complicated arithmetic processing is executed on a communication device (main device) installed at a remote place. To do. For example, Patent Document 1 proposes a technique related to a system that projects screen information of a main device (such as a personal computer or a server computer) onto a remote display device (display terminal) via a network.

In such a system, input information (such as pen input by a digitizer) from the display terminal is similarly transmitted to the main unit via the network, and the actual application program processing is executed by the main unit. Thereafter, the execution result and the screen update information are transferred to the display terminal via the network. The display terminal executes output processing (drawing processing) based on the received screen update information.

On the other hand, VNC (Virtual Network Computing) is known as a technique for efficiently transmitting screen information from a main unit on a remote network to a display terminal. In this VNC, when the update of the screen is detected, the value of the read pixel information is compared with the value of the pixel information transmitted to the previous display terminal, and the updated screen area changed from the previous time is determined. Further, after the still image compression of the updated screen area, only the compressed screen difference information is transmitted to the display terminal. Thereby, the consumption of a communication band can be suppressed. Accordingly, the amount of screen information to be transmitted increases when the screen change is large, such as movement of a window, and conversely, the amount of screen information to be transmitted decreases when the screen change is small.

Therefore, it is conceivable to improve the responsiveness regarding the display of the update screen by performing the decompression process of the compressed image received from the main unit by the display terminal with dedicated hardware. However, when processing in hardware, generally, the time required for hardware initialization processing, the time required for decompression processing of compressed images, the waiting time required for interrupt type notification of processing completion, etc. are required as overhead. . For this reason, depending on the size of the number of pixels of the image to be expanded, there may be a case where it takes less time to process with software than with hardware.

To solve this problem, the display terminal executes a decompression process by hardware when the number of pixels of the compressed image received from the main device is larger than a predetermined threshold value, and otherwise decompresses by software. A method of switching the processing so as to execute the processing can be considered.

As a method for selecting and implementing one process from a plurality of processes based on image information, in Patent Document 2, either hardware or software is selected based on still image addition (header) information. There has been proposed a technique for selecting and decompressing compressed data. Patent Document 3 proposes a technique of selecting and encoding either lossless compression or lossy compression based on image resolution information.

US Pat. No. 6,784,855 JP 2000-50263 A JP 2008-42681 A

However, as described above, the compressed image having the number of pixels larger than the threshold is subjected to decompression processing by hardware, and the compressed image having the number of pixels equal to or less than the threshold is switched by software so that the processing is switched. In some cases, the efficiency of image expansion processing on the terminal side could not be improved.

For example, when there are a plurality of update areas in which the number of pixels of the compressed image is smaller than the threshold value, it is more likely that the compressed image corresponding to the rectangular area obtained by integrating the plurality of update areas into one is expanded by hardware. In some cases, the processing time may be shorter than when the compressed image corresponding to the area is expanded by software a plurality of times. However, in the above method, when there are a plurality of update areas in which the number of pixels of the compressed image is smaller than the threshold value, it is only necessary to execute software decompression processing a plurality of times for each of the compressed images corresponding to the plurality of update areas. Can not.

The present invention has been made in view of the above, and provides a communication device and a display device capable of improving the efficiency of processing of a display device capable of decompressing a compressed image transmitted from a communication device by software and hardware. The purpose is to do.

The present invention is a communication device connectable to a display device capable of displaying an image via a network, an image storage unit for storing a display image to be displayed on the display device, and an update for updating the display image When an update image generation unit that generates an image, a detection unit that detects a difference region indicating a region where pixel information does not match between the update image and the display image, and a plurality of the difference regions are detected, A software processing time representing a processing time when software performs an expansion process on each compressed image obtained by compressing a plurality of the difference area images, and an integrated area image representing one area including the plurality of difference areas. A calculation unit for calculating a hardware processing time representing a processing time when the decompression process for the compressed image is executed by hardware; and the hardware process. A determination unit configured to determine whether or not time is smaller than the software processing time; and when the hardware processing time is determined to be smaller than the software processing time, a compressed image obtained by compressing the image of the integrated region is generated A compressed image generation unit and a transmission unit that transmits the generated compressed image to the display device are provided.

In addition, the present invention is a display device connected to a communication device via a network, and receives a compressed image obtained by compressing an image in the updated region and the number of pixels of the image in the updated region from the communication device. A receiving unit, a decompression circuit capable of executing decompression processing of the compressed image, a software decompression unit capable of executing decompression processing of the compressed image by software, the number of pixels and a predetermined threshold value are compared, When the number of pixels is smaller than the threshold, the software decompression unit decompresses the compressed image to generate an updated image corresponding to the updated area. When the number of pixels is equal to or greater than the threshold, the decompression circuit An image generation unit that expands the compressed image to generate a display image corresponding to the updated area, a display unit that displays the display image, and the software expansion unit. A measurement unit that measures a software processing time that represents the processing time of the decompression process and a hardware processing time that represents a processing time of the decompression process by the decompression circuit, the number of pixels of the image decompressed by the software decompression unit, and the software processing time The first function for calculating the software processing time from the number of pixels is generated based on the number of pixels, and the hardware processing time is calculated from the number of pixels based on the number of pixels of the image expanded by the expansion circuit and the hardware processing time. An information generation unit that generates a second function to be calculated, and a transmission unit that transmits the first function and the second function to the communication device.

According to the present invention, it is possible to improve the efficiency of processing of a display device that can decompress a compressed image transmitted from a communication device by software and hardware.

It is a block diagram which shows the structure of the communication system concerning this Embodiment. It is a block diagram of the communication apparatus concerning this Embodiment. It is a figure which shows an example of the relationship between the pixel number of an image, and the expansion process time. It is a block diagram of the display apparatus concerning this Embodiment. It is a figure which shows an example of the relationship between the pixel number of an image, and the expansion process time. It is a figure which shows an example of the transmission message for transmitting time information. It is a figure which shows an example of the transmission message for transmitting function information. It is a flowchart which shows the whole flow of the image transmission process in this Embodiment. It is a flowchart which shows the whole flow of the image display process in this Embodiment. It is a sequence diagram which shows the flow of the information communicated between a display apparatus and a communication apparatus. It is explanatory drawing which shows the hardware constitutions of the communication apparatus and display apparatus concerning this Embodiment.

DETAILED DESCRIPTION Exemplary embodiments of an apparatus, a method, and a program according to the present invention will be described below in detail with reference to the accompanying drawings.

The communication system according to the present embodiment includes a main device (communication device) that executes an application and a display terminal (display device) that displays a screen updated by executing the application. When the main device detects that there are a plurality of difference areas of image information generated by the operation of the application program, the main apparatus integrates the plurality of difference areas into one rectangular area, thereby executing an image executed on the display terminal It is determined whether the processing time of the decompression process is shortened. When it is determined that the processing time is shortened, the main body apparatus compresses the image information corresponding to the rectangular area obtained by integrating a plurality of difference areas and transmits the compressed image information to the display terminal. Thereby, the efficiency of the image expansion process at the display terminal can be improved.

FIG. 1 is a block diagram showing a configuration of a communication system according to the present embodiment. The communication system according to the present embodiment is a system that transmits an image of an updated part due to an event occurring on the screen of the main device to a display terminal. Hereinafter, such a communication system is referred to as a screen transfer system.

As shown in FIG. 1, a screen transfer system 10 according to the present embodiment includes a main body device 100 as a communication device, a radio base station 300 as an access point connected to the main body device 100 via a network 400, A display terminal 200 is provided as a display device that performs wireless communication with the wireless base station 300 through a wireless LAN.

The screen transfer system 10 has a function of wirelessly transferring a screen of application software operating on the main device 100 to the display terminal 200 via the radio base station 300 and displaying the application screen of the main device 100 on each display terminal 200. Have. In the screen transfer system 10, in order to transfer the screen updated on the main device 100 side to the display terminal 200 in real time, only the image information of the updated portion in the screen of the main device 100 is transferred. That is, the main device 100 can transmit the image information via the radio base station 300 to the display terminal 200 that displays the image information.

The display terminal 200 receives the image information from the main device 100, expands the received image information, and displays it on the corresponding part in the screen.

The wireless base station 300 is a wireless communication base station that complies with a wireless communication protocol such as IEEE802.11. The network 400 is a network that conforms to a wired communication protocol such as IEEE802.3, for example. The network form is not limited to this, and it may be configured to connect with another protocol. Further, the display terminal 200 and the main device 100 may be connected via a wired network.

Next, the detailed configuration of the main device 100 will be described with reference to FIG. FIG. 2 is a block diagram of main device 100 according to the present embodiment. As shown in the figure, the main apparatus 100 includes a display 101, an input device 102, an image buffer 121, a condition storage unit 122, a session information storage unit 123, an event acquisition unit 111, and a difference area detection unit 112. A compressed image generation unit 113, a region integration determination unit 114, a processing time calculation unit 115, a message analysis unit 116, a communication processing unit 117, and a session manager 118.

The display 101 is a display device realized by an LCD (Liquid Crystal Display) or the like. The input device 102 is realized by a mouse or the like that moves the cursor displayed on the screen of the display 101. In addition, as the input device 102, a keyboard, a trackball, or the like may be used.

The image buffer 121 is a storage unit that stores images. Based on the number of pixels of the image, the condition storage unit 122 performs processing time when the compressed image is decompressed by software on the display terminal 200 (hereinafter referred to as software processing time), and decompresses the compressed image by hardware on the display terminal 200. A predetermined condition for calculating the processing time (hereinafter referred to as hardware processing time) is stored. For example, the condition storage unit 122 has function information for deriving a function (first function) that outputs the software processing time with the number of pixels as an input, and a function (first operation) that outputs the hardware processing time with the number of pixels as an input. The function information for deriving (two functions) is stored as a condition.

For example, when the relationship between the number of pixels and software processing time or hardware processing time is expressed by a linear function, the slope information and intercept information of the linear function correspond to function information for deriving the function. The function is not limited to a linear function.

The session information storage unit 123 stores session information representing information regarding the display terminal 200 that is establishing a session with the main device 100. For example, the session information storage unit 123 includes user identification information for identifying a user, status information indicating whether the session is being used, and whether transmission control is TCP (Transmission Control Protocol) or UDP (User Datagram Protocol). Session information in which information such as transmission control information indicating whether or not there is associated is stored.

In a state where a plurality of display terminals 200 have established a session for one main device 100, session information storage unit 123 identifies a display terminal 200 that is a destination of a message transmitted from main device 100. Session information including identification information is stored.

The image buffer 121, the condition storage unit 122, and the session information storage unit 123 are configured by any commonly used storage medium such as an HDD (Hard Disk Drive), an optical disk, a memory card, and a RAM (Random Access Memory). can do.

The event acquisition unit 111 acquires an event generated by the operation of an application program or the like. For example, the event acquisition unit 111 includes an operating system (OS) that performs overall control of the computer, a virtual display driver having a function equivalent to a display driver incorporated in the OS, and application programs such as application software that operates on the OS. It is realized with.

The event acquisition unit 111 updates the screen (image) when the screen is updated by application software or when the cursor is moved by a mouse operation or the like to update an image of an arbitrary area in the screen. Get that as an event.

The event acquisition unit 111 includes an updated image generation unit 111a as a detailed configuration. The updated image generation unit 111a can be realized as, for example, a virtual display driver incorporated in the OS. When a screen update event is acquired, the update image generation unit 111a acquires a drawing command from the graphic engine of the OS, and generates a display image representing an image to be displayed on the display terminal 200 by performing a drawing process. The data is sequentially output to the buffer 121 and stored. As a result, display images are sequentially held in the image buffer 121.

In the following description, an image stored in the image buffer 121 is referred to as a display image, and an image that is newly generated by the update image generation unit 111a and is stored in the image buffer 121 is referred to as an update image. As described above, the update image generation unit 111a generates an update image to be displayed on the display terminal 200 according to an event generated by the operation of the application program.

The difference area detection unit 112 detects a difference area representing an area where the pixel information does not match between the old and new display images sequentially held in the image buffer 121. That is, the difference area detection unit 112, when notified by the update image generation unit 111a that an update image has been generated, generates new image information (update image) and the image buffered in the image buffer 121. A difference area between information (display image) is detected. The difference area detection unit 112 detects, for example, a minimum rectangle including a portion that does not match between two pieces of image information as a difference area. Note that the difference area detection unit 112 may be configured to check whether there is a difference at predetermined time intervals.

The compressed image generation unit 113 generates a compressed image obtained by compressing the difference region image detected by the difference region detection unit 112 for transmission. The compressed image may be generated using irreversible compression such as JPEG (Joint Photographic Experts Group) or may be generated using a lossless compression method.

Note that the compressed image generation unit 113 and the above-described difference area detection unit 112 are realized by an application program for screen transfer.

The region integration determination unit 114 determines whether or not it is more efficient for the display terminal 200 to integrate the plurality of difference regions when a plurality of difference regions are detected. For example, the region integration determination unit 114 receives the notification of the difference region information including the number of pixels of the image indicated by the difference region from the difference region detection unit 112, and the condition stored in the condition storage unit 122 and the notified number of pixels. Therefore, it is determined whether it is more efficient to integrate a plurality of difference areas.

As the number of pixels of the image indicated by the difference area, for example, a value obtained by multiplying the vertical width and the horizontal width of the image of the difference area is used. For example, when a rectangular area having a vertical width of 320 pixels and a horizontal width of 240 pixels is detected as the difference area, the difference area detection unit 112 sets the area integration determination unit as 76800 pixels obtained by multiplying both as the number of pixels of the image indicated by the difference area. 114 is notified.

Hereinafter, details of the determination processing by the region integration determination unit 114 will be described. First, the region integration determination unit 114 uses the processing time calculation unit 115 to integrate a plurality of difference regions into a single region (hereinafter referred to as an integrated region) and a software process for each of the plurality of difference regions. Calculate time.

That is, the processing time calculation unit 115 performs a hardware processing time when a compressed image obtained by compressing an image in the integrated area is decompressed by hardware and a compressed process obtained by compressing each compressed image obtained by compressing a plurality of difference area images by software. The software processing time is calculated.

Specifically, the processing time calculation unit 115 inputs the number of pixels of the plurality of difference area images notified from the difference area detection unit 112 to the first function stored in the condition storage unit 122, and sets the software processing time. calculate. In addition, the processing time calculation unit 115 inputs the number of pixels of the integrated region image obtained by integrating the plurality of difference regions detected by the difference region detection unit 112 into the second function stored in the condition storage unit 122, and performs hardware processing. Calculate time.

Then, the region integration determination unit 114 determines whether or not the calculated hardware processing time is smaller than the software processing time. If the calculated hardware processing time is small, it is determined that it is more efficient to integrate a plurality of difference regions, and the determination result is determined as a difference. The area detection unit 112 is notified. In addition, the compressed image generation unit 113 compresses the image of the integrated region obtained by integrating the plurality of difference regions when the region integration determining unit 114 described later determines that the integration of the plurality of difference regions is more efficient. Generate a compressed image.

Hereinafter, details of the determination processing by the region integration determination unit 114 will be further described with reference to FIG. FIG. 3 is a diagram illustrating an example of the relationship between the number of pixels of an image and the expansion processing time.

In the upper right of the figure, a state where two rectangular areas α and β are detected as difference areas to be updated on the display screen of the main device 100 is shown. The rectangular area 301 represents an integrated area obtained by integrating the rectangular areas α and β into one. The rectangular area 301 can be obtained as a minimum rectangular area including rectangular areas α and β, for example. An area γ indicates a rectangular area in the rectangular area 301 that is not actually updated.

Further, in the figure, a function (a relationship between the expansion processing time by the software expansion processing and the number of pixels when the horizontal axis is the number of pixels of the image and the vertical axis is the expansion processing time of the compressed image on the display terminal 200 ( An example of the first function) and a function (second function) representing the relationship between the medical processing time by the hardware expansion processing and the number of pixels is also shown.

In the figure, as a function representing the relationship between the decompression processing time by the software decompression processing and the number of pixels, a function expression representing the processing time when processing one rectangular area is represented by a solid line (SW (1 rectangle)) 2 A function equation representing the total processing time from the start to the end of processing when processing two rectangular areas separately is indicated by a dotted line (SW (2 rectangles)).

For example, when the display terminal 200 decompresses the rectangular regions α and β by software, the values of the decompression processing times corresponding to the horizontal axes “α” and “β” in the SW (1 rectangle) function formula are This is the expansion processing time for each rectangular area. When the rectangular areas α and β are processed separately, an overhead corresponding to the function intercept is required for each processing, and thus corresponds to the horizontal axis “α + β” in the SW (2 rectangles) function expression of FIG. The value of the expansion processing time is the total processing time when the two rectangular areas are processed by software.

On the other hand, the main device 100 can be integrated into one rectangular area 301 in the display screen by adding a non-updated area γ to the rectangular areas α and β. The main device 100 can calculate the processing time when the rectangular area 301 is expanded by hardware using the HW (one rectangle) function formula of FIG.

In the drawing, the expansion processing time corresponding to the horizontal axis “α + β + γ” in the function equation of HW (1 rectangle) is the expansion processing time corresponding to the horizontal axis “α + β” in the function equation of SW (2 rectangles). The case where it becomes shorter than it is illustrated. In such a case, the region integration determination unit 114 generates a compressed image of the plurality of rectangular regions separately when one rectangular region including two rectangular regions is cut out from the display screen and the compressed image is generated. It can be determined that the display terminal 200 is more efficient. As a result, the difference area detection unit 112 cuts out one rectangular area including a plurality of rectangular areas, and notifies the compressed image generation unit 113 of a target area for generating a compressed image.

In addition, although the figure showed the example when two rectangular areas were detected as a difference area, when more rectangular areas were detected, the intercept which added the offset for the number of detected difference areas The total processing time when each area is expanded separately by software is calculated from a function expression having information. Then, the calculated total processing time is compared with the processing time when it is integrated into one rectangular area and decompressed by hardware. If the hardware decompression time is shorter, the plurality of rectangular areas Is cut out from the display screen to generate a compressed image.

2, the message analysis unit 116 analyzes the message received from the display terminal 200. For example, when the time information including the time required for the decompression process of the compressed image is received from the display terminal 200, the message analysis unit 116 obtains the first function and the second function from the time information and stores them in the condition storage unit 122. . As described above, in the present embodiment, the time required for the actual decompression process is acquired from the display terminal 200, and a function for calculating the processing time can be dynamically obtained based on the acquired time. Instead of performing the above processing by the message analysis unit 116, a function may be obtained in advance and stored in the condition storage unit 122.

The communication processing unit 117 transmits / receives a message to / from an external device such as the display terminal 200. The communication processing unit 117 includes a transmission unit 117a that transmits a message and a reception unit 117b that receives the message. For example, the transmission unit 117 a transmits a transmission image message including the compressed image generated by the compressed image generation unit 113 to the display terminal 200.

The transmission image message includes a compressed image of the rectangular area to be updated and rectangular information. The compressed image indicates compressed still image information such as JPEG. The rectangular information is information indicating the drawing position of the image of the rectangular area to be updated. For example, the rectangular information indicates an image drawing position with a configuration such as (horizontal direction start point coordinates, vertical direction start point coordinates, horizontal width pixel number, vertical width pixel number).

The transmission unit 117a transmits a message to be transmitted via the radio base station 300 with the display terminal 200 specified by the session manager 118 described later as a destination.

Further, the receiving unit 117b receives a message from the display terminal 200 and passes it to the message analyzing unit 116. For example, the receiving unit 117b receives time information including the time required for decompression processing of the compressed image from the display terminal 200.

The session manager 118 manages communication (session) established with the display terminal 200. For example, when the session manager 118 establishes a session with a certain display terminal 200, the session manager 118 generates session information in which user identification information of the user of the display terminal 200, session state information, transmission control information, and the like are associated with each other. The information is stored in the information storage unit 123. Then, the session manager 118 specifies the display terminal 200 as a message communication destination with reference to the session information, and transmits / receives a message by the communication processing unit 117 with the specified display terminal 200 as the communication destination.

Next, the detailed configuration of the display terminal 200 will be described with reference to FIG. FIG. 4 is a block diagram of the display terminal 200 according to the present embodiment. As shown in the figure, the display terminal 200 includes a display 201, an input device 202, an antenna 203, an image buffer 221, a generation time storage unit 222, a session information storage unit 223, an input / output interface 211, A software decompression unit 212, a hardware decompression circuit 213, an image generation unit 214, a generation time measurement unit 215, a message generation unit 216, a wireless communication processing unit 217, and a session manager 218 are provided.

The display 201 is a display device realized by an LCD or the like. The input device 202 is realized by a digitizer or a touch screen that moves a cursor displayed on the screen of the display 201. The input information acquired by the input device 202 is passed to the input / output interface 211 (described later).

The antenna 203 transmits and receives radio waves for wireless communication with an external device such as the main device 100.

The image buffer 221 is a storage unit that stores images. Further, the generation time storage unit 222 stores the generation time (extension processing time) of the expanded image measured by the generation time measurement unit 215 described later. Specifically, the generation time storage unit 222 multiplies the method used for image expansion processing (either hardware expansion processing or software expansion processing) and the number of pixels of the image (the vertical width and the horizontal width of the compressed image). Value) and the time actually required for the expansion processing (expansion processing time) are stored in association with each other.

The session information storage unit 223 stores session information representing information regarding the main device 100 that is establishing a session. For example, the session information storage unit 223 stores session information including session state information and transmission control information.

Note that the image buffer 221, the generation time storage unit 222, and the session information storage unit 223 can be configured by any commonly used storage medium such as an HDD, an optical disk, a memory card, and a RAM.

The input / output interface 211 is an input / output interface for the display 201 and the input device 202, and is realized by an application program such as GUI (Graphical User Interface).

For example, the input / output interface 211 acquires image information from the image buffer 221 and displays it on the display 201. Further, the input / output interface 211 acquires the image information sent from the main body device 100 via the image generation unit 214 and writes it to the image buffer 221, in addition to the function for GUI generated independently in the display terminal 200. The image information is written into the image buffer 221.

The software decompression unit 212 performs decompression processing of the compressed image received from the main device 100 by software in accordance with an instruction from the image generation unit 214 described later. The hardware decompression circuit 213 executes the decompression processing of the compressed image received from the main body device 100 by hardware according to the instruction from the image generation unit 214.

The image generation unit 214 expands the compressed image received from the main device 100 by the software expansion unit 212 or the hardware expansion circuit 213 and then writes the expanded image information to the designated drawing position of the image buffer 221 for drawing. That is, the image generation unit 214 displays a partial image generated by expanding the compressed image transmitted from the main device 100 and received by the wireless communication processing unit 217 at a designated position on the display 201.

The image generation unit 214 performs software expansion processing (expansion processing by the software expansion unit 212) and hardware expansion processing (by the hardware expansion circuit 213) based on the size of the number of pixels of the compressed image and a predetermined threshold. Switch between (decompression processing). Note that the image generation unit 214 can acquire the number of pixels of the compressed image from the rectangular information in the transmission image message or the number of pixels (resolution) information in the header of the compressed image.

For example, the threshold value is a rectangular area of 10,000 pixels obtained by multiplying the vertical width by 100 pixels and the horizontal width by 100 pixels. For example, if the received compressed image is 76800 pixels obtained by multiplying the vertical width by 320 pixels and the horizontal width by 240 pixels, the number of images is larger than the threshold value, so the image generation unit 214 uses the hardware expansion circuit 213 to The compressed image is decompressed by the hardware.

Further, for example, if the received compressed image is 1024 pixels obtained by multiplying the horizontal width by 32 pixels and the horizontal width by 32 pixels, the number of images is smaller than the threshold value. Decompress the compressed image.

Here, a method for switching the expansion process by the image generation unit 214 will be further described with reference to FIG. FIG. 5 is a diagram illustrating an example of the relationship between the number of pixels of an image and the expansion processing time.

This figure shows a functional equation with the resolution (number of pixels) as the horizontal axis and the expansion processing time as the vertical axis. “SW” represents the relationship between the number of pixels and the expansion process time when the expansion process is performed by software. “HW” represents the relationship between the number of pixels and the expansion processing time when the expansion processing is performed by hardware.

When performing decompression processing by hardware, the waiting time until handling hardware initialization processing and decompression processing completion notification, specifically waiting time from hardware device to interrupt notification to operating system is fixed. Exist as overhead. For this reason, the intercept value of the function becomes large as shown in FIG. On the other hand, when decompression processing by software is performed, such an overhead problem is solved, so that the intercept value of the function becomes small.

Also, the processing time focused on the portion where the compressed image is expanded is shorter when processed by hardware than when processed by software. For this reason, the slope of the function formula indicating the relationship between the processing time by hardware and the number of pixels is smaller than the slope of the function formula by software. Therefore, it is more efficient to execute the decompression process with software when the number of pixels of the image is small, and it is more efficient to execute the extension process with hardware when the number of pixels of the image is large.

In view of such a situation, the image generation unit 214 may set the pixel value corresponding to the intersection portion where the two function formulas intersect as a threshold when switching between software expansion processing and hardware expansion processing. desirable.

The information for deriving the function as shown in the figure is statistical information on the type of decompression processing, the number of pixels of the decompressed image, the decompression processing time, etc. while actually decompressing the compressed image on the display terminal 200. Can be obtained and stored in the generation time storage unit 222. A method of storing predetermined statistical information in the generation time storage unit 222 of the display terminal 200 or the condition storage unit 122 of the main device 100 may be used.

2, the generation time measurement unit 215 measures the decompression processing time by the image generation unit 214 and stores it in the generation time storage unit 222. Specifically, the generation time measurement unit 215 associates the method used by the image generation unit 214 for decompression processing of the compressed image (decompression method), the number of pixels of the decompressed image, and the decompression processing time, and generates the generation time. Save in the storage unit 222.

The message generation unit 216 generates a message to be transmitted to an external device such as the main device 100. For example, the message generation unit 216 transmits time information in which the expansion method, the number of pixels of the expanded image, and the expansion processing time are associated with each other from the information stored in the generation time storage unit 222. Generate as a message.

FIG. 6 is a diagram illustrating an example of a transmission message for transmitting time information. FIG. 6 shows a configuration example of a transmission message including the decompression processing time of the compressed image actually measured by the display terminal 200. As shown in the figure, the time information transmission message includes an IP (Internet Protocol) header, a UDP / TCP header, a message type, an expansion method, area information, and an expansion processing time.

The IP header is used to identify the destination address of the transmission / reception terminal. The UDP / TCP header indicates transmission control protocol information. The message type is used for identifying that the message is a message including time information generated by the display terminal 200. The expansion method is information for identifying whether the expansion processing time indicates the software processing time or the hardware processing time. The area information includes the number of pixels of the expanded image. The expansion processing time describes the processing time of the actually measured expansion processing in units of milliseconds, for example. Note that the information from the expansion method to the expansion processing time is added by N pieces measured by the display terminal 200.

As described above, when the message analysis unit 116 of the main device 100 receives a transmission message including time information as shown in FIG. 6, the number of pixels and the expansion processing time are calculated from the time information obtained by analyzing the transmission message. A function expression indicating the relationship between and is obtained. That is, when the message configuration shown in FIG. 6 is used, the main body apparatus 100 does not derive the function expression but the display terminal 200.

On the other hand, the display terminal 200 can also be configured to obtain a function expression from the time information and transmit the function information representing the obtained function expression to the main device 100. FIG. 7 is a diagram illustrating an example of a transmission message for transmitting function information from the display terminal 200 to the main device 100 when configured in this manner.

FIG. 7 shows the slope information and intercept information for deriving the first function that represents the relationship between the number of pixels and the software processing time, and the second function that represents the relationship between the number of pixels and the hardware processing time. The example of a structure of the transmission message containing inclination information and intercept information is shown. The expansion method is information for identifying whether the expansion processing time indicates the software processing time or the hardware processing time. The inclination information indicates the inclination of a functional expression indicating the relationship between the number of pixels and the expansion processing time as shown in FIG. The intercept information indicates the intercept of the functional expression. As shown in FIG. 7, two pieces of information corresponding to software processing and hardware processing are added to the information from the decompression method to the intercept information.

Note that the message generation unit 216 receives the generated time information (or function information) via the wireless communication processing unit 217 when, for example, a communication session with the main device 100 is started or at regular time intervals. To 100.

2, the radio communication processing unit 217 transmits and receives signals to and from the radio base station 300 via the antenna 203. The wireless communication processing unit 217 includes a transmission unit 217a that transmits a message and a reception unit 217b that receives the message. The wireless communication processing unit 217 is realized by a wireless LAN function compliant with IEEE 802.11 or the like.

For example, the reception unit 217b of the wireless communication processing unit 217 demodulates the received wireless signal to generate a packet, and passes the corresponding data to the image generation unit 214 according to the message type of the packet. For example, when the packet is a packet of a transmission image message including a compressed image, information such as the compressed image and the number of pixels extracted from the packet is passed to the image generation unit 214.

For example, when the input device 202 acquires input information from the user, the input / output interface 211 analyzes the coordinate information from the input information and then transmits the input information to the main body device 100 via the transmission unit 217a of the wireless communication processing unit 217. Sent. In this case, the main device 100 executes application processing based on the input information received from the display terminal 200. When an update occurs in the screen area as a result of the application process, the main device 100 acquires the image information to be updated, generates a transmission image message, and transmits the message to the display terminal 200. Also in this case, on the main device 100 side, when there are a plurality of difference areas of the detected image, it is more efficient at the time of expansion processing of the display terminal 200 to integrate the plurality of difference areas into one rectangular area. A transmission image message is generated after determining whether to obtain the transmission image message.

The session manager 218 manages communication (session) established with the main device 100. For example, when the session manager 218 establishes a session with the main device 100, the session manager 218 generates session information in which session state information, transmission control information, and the like are associated with each other, and stores the session information in the session information storage unit 223.

Next, image transmission processing by the main body apparatus 100 according to the present embodiment configured as described above will be described with reference to FIG. FIG. 8 is a flowchart showing the overall flow of the image transmission process in the present embodiment.

First, the difference area detection unit 112 performs image information (display image) stored in the image buffer 121 and new image information (update image) generated by the update image generation unit 111a at predetermined time intervals. A difference area is detected (step S801).

When the difference area is detected, the area integration determination unit 114 determines whether or not a plurality of difference areas are detected (step S802). When a plurality of difference areas are detected (step S802: YES), the processing time calculation unit 115 calculates an extension processing time (software processing time) when each of the plurality of difference areas is extended by software (step S803). ). For example, if two difference areas are detected, the processing time calculation unit 115 corresponds to the number of pixels of the images of the plurality of difference areas by using the SW (2 rectangle) function formula of FIG. Calculate software processing time.

Next, the processing time calculation unit 115 calculates an expansion processing time (hardware processing time) when one integrated region obtained by integrating a plurality of difference regions is extended by hardware (step S804). For example, the processing time calculation unit 115 calculates the hardware processing time corresponding to the number of pixels of the image in the integrated region by using the function formula of HW (one rectangle) in FIG.

Next, the region integration determination unit 114 determines whether or not the calculated hardware processing time is smaller than the calculated software processing time (step S805). If the hardware processing time is smaller than the software processing time (step S805: YES), the difference area detection unit 112 sets the integrated area obtained by integrating a plurality of difference areas as a target area for generating a compressed image to the compressed image generation unit 113. Notification is made (step S806).

Next, the compressed image generation unit 113 generates a compressed image obtained by compressing the image in the integrated area (step S807). If a plurality of difference areas are not detected in step S802 (step S802: NO), the compressed image generation unit 113 generates a compressed image obtained by compressing the detected image of one difference area. If it is determined in step S805 that the hardware processing time is not shorter than the software processing time (step S805: NO), the compressed image generation unit 113 individually compresses the detected plurality of difference regions. To generate a plurality of compressed images.

Next, the transmission unit 117a transmits a transmission image message including the compressed image generated by the compressed image generation unit 113 and the number of pixels of the compressed image to the display terminal 200 (step S808), and the image transmission process is terminated.

Next, image display processing by the display terminal 200 according to the present embodiment configured as described above will be described with reference to FIG. FIG. 9 is a flowchart showing the overall flow of the image display processing in the present embodiment.

First, the receiving unit 217b of the wireless communication processing unit 217 receives a transmission image message including the compressed image and the number of pixels sent from the main device 100 (step S901). The reception unit 217b extracts the compressed image and the number of pixels from the transmission image message, and notifies the image generation unit 214 of them.

Next, the image generation unit 214 determines whether or not the notified number of pixels of the compressed image is smaller than a predetermined threshold (step S902). When the number of pixels is smaller than the threshold (step S902: YES), the image generation unit 214 decompresses the compressed image by software using the software decompression unit 212 (step S903). If the number of pixels is equal to or greater than the threshold (step S902: NO), the image generation unit 214 decompresses the compressed image by hardware using the hardware decompression circuit 213 (step S904).

Next, the image generation unit 214 draws the image information generated by the decompression process at the designated position of the image buffer 221 via the input / output interface 211 (step S905). As a result, the updated portion is reflected on the display 201.

Next, the flow of information communicated between the display terminal 200 and the main device 100 will be described with reference to the sequence diagram of FIG. In the example shown in the figure, first, the message generation unit 216 of the display terminal 200 transmits time information including the decompression processing time of the compressed image by software and hardware to the main device 100 at the start of the communication session. (Step S1001).

When receiving the time information from the display terminal 200, the message analysis unit 116 of the main device 100 obtains the slope information and intercept information (function information) of the linear function for calculating the expansion processing time from the received time information, and the condition The function information stored in the storage unit 122 is updated.

After that, the difference area detection unit 112 of the main device 100 detects an update part (difference area) in the display screen according to the result of the application process (step S1002). The region integration determination unit 114 performs integration determination of a plurality of difference regions when a plurality of difference regions are detected (step S1003). Based on the result, the compressed image generation unit 113 performs generation of a compressed image and transmission processing of a transmission image message including the compressed image (step S1004). As a result, a compressed image obtained by compressing the image in the difference area is transmitted to the display terminal 200 (step S1005). The time information may be implemented not only at the start of the communication session but also at regular time intervals.

As described above, according to the communication apparatus (main apparatus 100) according to the present embodiment, the main apparatus 100 can generate and transmit a compressed image having the optimum size for the display apparatus (display terminal 200). Thus, it is possible to improve the efficiency of the processing on the display terminal 200 side, which is performed by switching between the decompression process by software and the decompression process by hardware.

Next, the hardware configuration of the communication device (main device 100) and the display device (display terminal 200) according to the present embodiment will be described with reference to FIG. FIG. 11 is an explanatory diagram illustrating a hardware configuration of the communication device and the display device according to the present embodiment.

The communication device and the display device according to the present embodiment communicate with a control device such as a CPU (Central Processing Unit) 51 and a storage device such as a ROM (Read Only Memory) 52 and a RAM 53 while communicating with a network. An I / F 54, an external storage device such as an HDD (Hard Disk Drive), a CD (Compact Disc) drive device, a display device such as a display device, an input device such as a keyboard and a mouse, and a bus 61 that connects each part. I have.

The image transmission program executed by the communication device according to the present embodiment and the communication program executed by the display device are files in an installable format or an executable format, CD-ROM (Compact Disk Read Only Memory), flexible It is recorded on a computer-readable recording medium such as a disk (FD), CD-R (Compact Disk Recordable), DVD (Digital Versatile Disk) and the like.

In addition, the image transmission program executed by the communication device according to the present embodiment and the communication program executed by the display device are stored on a computer connected to a network such as the Internet, and provided by being downloaded via the network. You may comprise. The image transmission program executed by the communication apparatus according to the present embodiment and the communication program executed by the display apparatus may be provided or distributed via a network such as the Internet.

Further, the image transmission program and the communication program according to the present embodiment may be provided by being incorporated in advance in a ROM or the like.

The image transmission program executed by the communication apparatus according to the present embodiment includes the above-described units (event acquisition unit, difference region detection unit, compressed image generation unit, region integration determination unit, processing time calculation unit, message analysis unit, communication). As a real hardware, the CPU 51 (processor) reads the image transmission program from the storage medium and executes it to load the above-described units onto the main storage device. The above-described units are generated on the main storage device.

The communication program executed by the display device according to the present embodiment includes the above-described units (input / output interface, software expansion unit, hardware expansion circuit, image generation unit, generation time measurement unit, message generation unit, wireless communication). It is a module configuration including a processing unit and a session manager), and as actual hardware, the CPU 51 (processor) reads the communication program from the storage medium and executes it to load each unit on the main storage device, Each unit described above is generated on the main memory.

As described above, the apparatus, method, and program according to the present invention are suitable for an apparatus that realizes a function of sharing an application screen between apparatuses.

51 CPU
52 ROM
53 RAM
54 Communication I / F
61 Bus 10 Screen Transfer System 100 Main Device 101 Display 102 Input Device 111 Event Acquisition Unit 111a Update Image Generation Unit 112 Difference Region Detection Unit 113 Compressed Image Generation Unit 114 Region Integration Determination Unit 115 Processing Time Calculation Unit 116 Message Analysis Unit 117 Communication Processing Unit 117a transmission unit 117b reception unit 118 session manager 121 image buffer 122 condition storage unit 123 session information storage unit 200 display terminal 201 display 202 input device 203 antenna 211 input / output interface 212 software expansion unit 213 hardware expansion circuit 214 image generation unit 215 Generation time measurement unit 216 Message generation unit 217 Wireless communication processing unit 217a Transmission unit 217b Reception unit 218 Session money Catcher 221 image buffer 222 generating time storage unit 223 the session information storage unit 300 radio base station 301 rectangular regions 400 Network

Claims

A communication device connectable to a display device capable of displaying an image via a network,
An image storage unit for storing a display image to be displayed on the display device;
An update image generation unit for generating an update image for updating the display image;
A detection unit for detecting a difference region indicating a region where pixel information does not match between the updated image and the display image;
When a plurality of difference areas are detected, a software processing time representing a processing time when the decompression process for each compressed image obtained by compressing the images of the plurality of difference areas is executed by software; and the plurality of difference areas A calculation unit that calculates a hardware processing time that represents a processing time when the decompression process for the compressed image obtained by compressing the image of the integrated region that represents one area including
A determination unit for determining whether the hardware processing time is smaller than the software processing time;
A compressed image generation unit that generates a compressed image obtained by compressing the image of the integrated region when it is determined that the hardware processing time is smaller than the software processing time;
A transmission unit for transmitting the generated compressed image to the display device;
A communication apparatus comprising:
A condition storage unit that stores a predetermined condition for calculating the software processing time and the hardware processing time from the number of pixels of an image;
The calculation unit calculates the software processing time using the condition from the number of pixels of each of the plurality of difference area images, and calculates the hardware processing time using the condition from the number of pixels of the integrated area image. To calculate,
The communication apparatus according to claim 1.
The condition storage unit includes a first function that outputs the software processing time as a result of inputting the number of pixels of an image, and a second function that outputs the hardware processing time as a result of inputting the number of pixels of an image. Remember as a condition,
The calculation unit calculates the software processing time by inputting the number of pixels of the image of the plurality of difference areas to the first function, and the result of inputting the number of pixels of the image of the integration area to the second function Calculating the hardware processing time output as
The communication device according to claim 2.
Reception that receives from the display device time information that associates a decompression method that indicates whether the decompression process is performed by software or hardware, the number of pixels of the decompressed image, and the processing time required for the decompression process And
The first function is generated from the time information including the decompression method representing that the decompression process has been executed by software, and the second function is derived from the time information including the decompression method representing that the decompression process has been performed by hardware. A message analysis unit that generates a function and stores the generated first function and the second function as the conditions in the condition storage unit;
The communication device according to claim 3.
A receiving unit for receiving the first function and the second function from the display device;
A message analysis unit that stores the received first function and the second function as the conditions in the condition storage unit;
The communication device according to claim 4.
The detection unit detects the difference area representing a rectangular area that does not match between the update image and the display image;
When the plurality of difference areas are detected, the calculation unit executes the software processing time and the image processing for the integrated area representing the smallest rectangular area including the plurality of difference areas by hardware. Calculating the hardware processing time;
The communication device according to claim 5.
A display device connected to a communication device via a network,
A receiving unit that receives the compressed image obtained by compressing the image of the updated region and the number of pixels of the image of the updated region from the communication device;
A decompression circuit capable of executing decompression processing of the compressed image;
A software decompression unit capable of executing decompression processing of the compressed image by software;
Comparing the number of pixels with a predetermined threshold, and when the number of pixels is smaller than the threshold, decompress the compressed image by the software decompression unit to generate an updated image corresponding to the updated region, An image generation unit that generates a display image corresponding to the updated region by expanding the compressed image by the expansion circuit when the number of pixels is equal to or greater than the threshold;
A display unit for displaying the display image;
A measurement unit that measures a software processing time that represents the processing time of the decompression process by the software decompression unit and a hardware processing time that represents the processing time of the decompression process by the decompression circuit;
A time associated with a decompression method indicating whether the decompression process is performed by the software decompression unit or the decompression circuit, the number of pixels of the decompressed image, and either the software processing time or the hardware processing time An information generator for generating information;
A transmission unit for transmitting the time information to the communication device;
A display device comprising:
A display device connected to a communication device via a network,
A receiving unit that receives the compressed image obtained by compressing the image of the updated region and the number of pixels of the image of the updated region from the communication device;
A decompression circuit capable of executing decompression processing of the compressed image;
A software decompression unit capable of executing decompression processing of the compressed image by software;
Comparing the number of pixels with a predetermined threshold, and when the number of pixels is smaller than the threshold, decompress the compressed image by the software decompression unit to generate an updated image corresponding to the updated region, An image generation unit that generates a display image corresponding to the updated region by expanding the compressed image by the expansion circuit when the number of pixels is equal to or greater than the threshold;
A display unit for displaying the display image;
A measurement unit that measures a software processing time that represents the processing time of the decompression process by the software decompression unit and a hardware processing time that represents the processing time of the decompression process by the decompression circuit;
A first function for calculating the software processing time from the number of pixels is generated based on the number of pixels of the image expanded by the software expansion unit and the software processing time, and the number of pixels of the image expanded by the expansion circuit and the hardware An information generation unit that generates a second function that calculates the hardware processing time from the number of pixels based on the hardware processing time;
A transmitter that transmits the first function and the second function to the communication device;
A display device comprising: