WO2014057809A1

WO2014057809A1 - Motion video transmission system and method

Info

Publication number: WO2014057809A1
Application number: PCT/JP2013/075965
Authority: WO
Inventors: 一範小澤
Original assignee: 日本電気株式会社
Priority date: 2012-10-12
Filing date: 2013-09-18
Publication date: 2014-04-17

Abstract

A transmission-side device segments a screen into a plurality of blocks, and transmits as either image data which encodes the block or a render command which renders the block. With a receiving-side device, an image signal of a block is generated by rendering according to either the decoding of the image data or the render command.

Description

Video transmission system and method

The present invention relates to transmission of moving images over a data communication network. In particular, in a thin client system in which a terminal such as a mobile terminal accesses a server remotely via a network such as a mobile network and operates a virtual client on the server, the screen of the virtual client is transmitted from the server to the terminal. It relates to the transmission of moving images.

In recent years, the use of thin clients has begun to spread in order to ensure a high level of security, particularly in companies. This is a technology that operates a virtual client on the server as if operating a real terminal from a terminal, operates an application on the virtual terminal to generate screen information, transfers the screen information to the terminal, and displays it on the terminal. In addition, since no data is left in the terminal, there is an advantage that even if the terminal is lost, there is no possibility that secret information or company information is leaked to the outside.
There is a thin client system based on a system in which an application is operated on the server side to generate a screen, the screen is compressed and transferred to a terminal, and the screen is decoded and displayed on the terminal. Some systems of this type have a configuration in which a screen is always compressed and encoded using an image codec regardless of the screen status. In such a configuration, the amount of data per unit time transferred from the server to the terminal is approximately proportional to the screen size of the terminal. For this reason, when the network bandwidth is narrow relative to the screen size of the terminal, data transfer is likely to be delayed, resulting in delays in screen update on the terminal side and reaction delays on the terminal side. May occur. In particular, in a system in which a terminal and a server are connected via a network including a wireless zone, such as a mobile communication terminal such as a mobile phone, the time variation of the network bandwidth is likely to increase and the above-described delay is likely to occur. .
On the other hand, a method is also known in which a screen drawing command for generating a screen at a terminal is transferred from the server without transferring the screen itself from the server. In the case of this method, when the screen movement is small, the amount of data transferred from the server to the terminal can be reduced compared to the former case, but when the screen is complicated and the movement is large, the data is transferred from the server to the terminal. Since the number of commands is greatly increased, the processing load on the terminal is greatly increased compared to the former in order to draw the screen on the terminal. For this reason, the processing time in a terminal increases and responsiveness falls. Also, the amount of power used by the terminal increases due to an increase in processing load at the terminal.
Patent Document 1 is cited as a document describing a technique related to the present invention. Patent Document 1 describes a server that instructs another client device to transmit drawing information for updating the screen of one client device.
Patent Document 2 is cited as a document in which other techniques related to the present invention are described. Japanese Patent Application Laid-Open No. 2004-228561 describes a method in which a screen is divided into a plurality of blocks, compression-coded for each block, and transmitted to a client device with priority from a block having a small image size after compression coding.

JP 2012-079222 A JP 2008-211639 A

The present invention has been made in view of such a situation, and the problem to be solved by the present invention is based on the communication status of the network and the nature of the image to be transmitted when transmitting an image constituting a moving image. Accordingly, it is possible to perform transmission in an appropriate transmission form.

In order to solve the above-described problems, the present invention has, as one aspect thereof, a transmission-side device that transmits a moving image for each screen via a data communication network and a reception-side device that receives the screen via the data communication network. And the transmission side device is any one of means for dividing the screen into a plurality of blocks, image data obtained by encoding the block for each of the plurality of blocks, and a drawing command for drawing the block Selecting means, means for encoding a block into image data, means for determining a drawing command for drawing the block from the block, and receiving the image data and drawing command via the data communication network Means for transmitting to the side device, the receiving side device from the transmitting side device to the data communication network. Means for discriminating the image data and drawing command received via the network, means for decoding the image data and outputting as an image signal constituting part of the screen, and part of the screen according to the drawing command There is provided a moving image transmission system comprising means for outputting an image signal of an image drawn.
Further, the present invention provides, as another aspect, means for dividing one screen of a moving image into a plurality of blocks, image data obtained by encoding the block for each of the plurality of blocks, and a drawing command for drawing the block A means for selecting any one of the above, a means for encoding the block into image data, a means for determining a drawing command for drawing the block from the block, and the image data and the drawing command via the data communication network. A data communication apparatus comprising means for transmitting to another data communication apparatus is provided.
In another aspect, the present invention corresponds to one of a plurality of blocks obtained by dividing one screen of a moving image, and an image received from the other data communication apparatus via the data communication network. Means for discriminating data or a drawing command, means for decoding the image data and outputting it as an image signal constituting a part of the screen, and an image signal of an image obtained by drawing a part of the screen according to the drawing command Provided is a data communication device comprising a means for outputting.
Further, the present invention provides, as another aspect, means for dividing one screen of a moving image into a plurality of blocks, image data obtained by encoding the block for each of the plurality of blocks, and a drawing command for drawing the block A means for selecting any one of the above, a means for encoding the block into image data, a means for determining a drawing command for drawing the block from the block, and the image data and the drawing command via the data communication network. A program for causing a computer to function as means for transmitting to another data communication apparatus is provided.
According to another aspect of the present invention, as a further aspect, the transmitting side device divides one screen of the moving image into a plurality of blocks, and the transmitting side device uses the image data of the block for one of the plurality of blocks. Selecting any one of encoding to and determination of a drawing command for drawing the block, executing one of the above in response to the selection at the transmitting side device, the image data, and A step of transmitting any of the drawing commands from the transmission side device to the reception side device via the data communication network, and the reception side device according to the decoding of the image data and the drawing command A moving image transmission method comprising the step of generating an image signal constituting a part of the screen by performing any one of the drawn images.

According to the present invention, when one block of a screen is transmitted, it is transmitted as either image data or a drawing command, so that the block can be transmitted in a transmission form according to the situation.

FIG. 1 is a block diagram of a moving picture transmission system 1 according to the first embodiment of the present invention.
FIG. 2 is a block diagram of a thin client system 100 according to the second embodiment of the present invention.
FIG. 3 is a block diagram of the server device 110 of the thin client system 100.
FIG. 4 is a block diagram of the determination unit 185 of the server device 110.
FIG. 5 is a block diagram of the image encoding unit 186 of the server apparatus 110.
FIG. 6 is a block diagram of the portable terminal 170 of the thin client system 100.

A moving picture transmission system 1 according to a first embodiment of the present invention will be described with reference to FIG. The moving image transmission system 1 is a system that transmits a moving image for each screen from the transmission side device 3 to the reception side device 4 via the network 2. The network 2 is a data communication network, and is particularly suitable for a network whose communication status changes according to time, such as a mobile communication network such as a mobile phone network as at least a part thereof.
The transmission side device 3 is a device that transmits a time-changing screen to the reception side device 4. A server of a thin client system as in the second embodiment to be described later, that is, a server that executes a virtual client and provides a screen of the virtual client to the thin client is a preferable example of the transmission side device 3. You may use this invention for transmission of the screen produced | generated by means other than a thin client server. For example, it may be an information processing apparatus having a moving image shooting function and a data communication function, more specifically, a mobile phone terminal having a moving image shooting function, or a digital video camera having a wireless communication function.
The transmission side device 3 includes a dividing unit 5, a determination unit 6, an image encoder 7, a drawing command encoder 8, and a transmission unit 9. The dividing unit 5 divides a screen input from a moving image photographing function (not shown) into a plurality of areas, for example, m × n (m and n are natural numbers) areas. Hereinafter, each divided area is referred to as a block. The discriminating unit 6 selects one of the image encoder 7 and the drawing command encoder 8 according to the block itself input from the dividing unit 5 and the communication status of the network 2, and passes the block to the selected encoder. The image encoder 7 encodes the transferred block into image data of a predetermined format. The image encoder 7 may encode the block according to any of a plurality of types of compression encoding methods. The drawing command encoder 8 obtains a drawing command group necessary for drawing the input block, and encodes the obtained drawing command group according to a predetermined compression encoding method. The transmission unit 9 transmits the block encoded by either the image encoder 7 or the drawing command encoder 8 to the reception side device 4 via the network 2.
The receiving side device 4 is a device that receives and displays a screen from the transmitting side device 3. The terminal of the thin client system of the second embodiment to be described later, that is, a terminal that receives a virtual client screen from a server that executes the virtual client, particularly a mobile communication terminal such as a mobile phone terminal, Although it is a suitable example, it may be applied to other devices that receive and display a moving image from a network, such as an information processing device, a workstation, a server, or a personal computer having a data communication function, and a moving image display function. It may be a mobile phone terminal or other mobile information processing device equipped with
The receiving side device 4 includes a receiving unit 10, an image decoder 11, a drawing command decoder 12, a screen display unit 13, and a screen drawing unit 14. When receiving the encoded block from the transmitting unit 9 via the network 2, the receiving unit 10 determines whether the encoded data is encoded as image data of a predetermined format or the drawing command group, and the former In this case, the image is output to the image decoder 11, and in the latter case, the image is output to the drawing command decoder 12. The image decoder 11 decodes the image data according to the encoding method used by the image encoder 7 for encoding, and displays the image data in a corresponding area on the screen of the screen display unit 13. The corresponding area is an area occupied by the block in the screen before the screen including the block corresponding to the image data is input to the dividing unit 5. The drawing command decoder 12 decodes the encoded drawing command group and outputs it. The screen drawing unit 14 draws an image in a corresponding area on the screen of the screen display unit 13 in accordance with the input drawing command group. The screen display unit 13 is a display device, and specifically, there are a liquid crystal display device, a plasma display, a cathode ray tube, and the like, but the type thereof is not limited.
In a configuration in which a block is transmitted only as image data regardless of the block or communication status as in the past, for example, even a block that can be drawn with a simple drawing command group is transmitted as image data. The amount of data increases, and the display on the screen display unit 13 may be delayed depending on the communication status of the network. On the other hand, regardless of the block and communication status, in the configuration in which the block is transmitted only as a drawing command, the number of commands is greatly increased when converting a block of a screen that is particularly complex and intensely moving into a drawing command. Not only does the amount of data to be transmitted increase, but also the processing time required to execute all the increased drawing commands in the screen drawing unit 14 becomes longer. On the other hand, in this embodiment, the block is encoded using either the image encoder 7 or the drawing command encoder 8 according to the communication status of the network 2 and the block itself to be transmitted. Blocks can be transmitted in a transmission form suitable for the situation, and display delay on the screen display unit 13 can be suppressed.
A thin client system 100 according to a second embodiment of the present invention will be described with reference to FIG. In this embodiment, an example is shown in which a mobile network 150 is used as the network. Moreover, the structure in the case of using a SGSN / GGSNN apparatus as a packet transfer apparatus is shown. Here, the SGSN / GGSN device means a device in which an SGSN (Serving GPRS Support Node) device and a GGSN (Gateway GPRS Support Node) device are integrated. In FIG. 2, as an example, the thin client server device 110 is arranged on the cloud network 130 and the cloud network 130 and the mobile network 150 are connected.
In FIG. 2, the end user connects the mobile terminal 170 to the virtual client of the server apparatus 110 arranged in the cloud network 130 and operates the virtual client as if operating the real terminal. For this purpose, the client software of the mobile terminal 170 transmits a packet storing the operation signal to the server device 110 via the base station 194, the RNC device 195, and the SGSN / GGSN device 190 on the mobile network 150. Here, the operation signal means a signal transmitted from the mobile terminal 170 to the server device 110 by an operation such as a key operation on the mobile terminal 170, a touch operation on the screen, character input, or scrolling.
The operation signal packet transmitted from the packet transmission unit in the client software installed in the mobile terminal 170 is transmitted to the cloud network 130 via the base station device 194, the RNC device 195, and the SGSN / GGSN device 190 on the mobile network 150. The server device 110 receives the operation signal. Here, a well-known protocol can be used for sending the operation signal, but UDP / IP is used, but TCP / IP or the like can also be used.
FIG. 3 is a block diagram illustrating a configuration of the server device 110. The operation signal packet receiving unit 182 receives a packet storing the operation signal from the mobile terminal 170 via the base station 194, the RNC device 195, and the SGSN / GGSN device 190. The operation signal packet receiving unit 182 extracts an operation signal from the received operation signal UDP / IP packet and outputs the operation signal to the virtual client unit 211.
The virtual client unit 211 includes application software corresponding to various services, a control unit, a screen generation unit, a cache memory, and the like. In addition, the application software can be easily updated from the outside of the server device 110. The virtual client unit 211 analyzes the operation signal input from the operation signal packet receiving unit 182, activates the application software specified by the operation signal, and displays a screen drawn by the application software and the OS as a predetermined screen. The image is generated at the resolution and output to the screen capture unit 180. Further, a drawing command executed when generating and drawing the screen is output to the drawing command collecting unit 181.
The drawing command collection unit 181 collects the drawing command group output from the virtual client unit 211 for each screen, temporarily saves it for each screen, and outputs it to the determination unit 185.
The screen capture unit 180 captures and outputs a screen at a predetermined screen resolution and frame rate.
The dividing unit 184 divides the captured screen into a plurality of blocks having a predetermined size. Here, the block size is, for example, 16 pixels × 16 lines, but other sizes, for example, 8 pixels × 8 lines, 4 pixels × 4 lines, and the like can be used. The smaller the block size, the better the discrimination accuracy in the discriminator, but the processing amount increases. The dividing unit 184 outputs the divided blocks to the determining unit 185.
The configuration of the determination unit 185 is shown in FIG. Here, in this embodiment, the determination unit 185 determines whether to transfer the screen based on at least one of the feature amount obtained from the image signal of the screen and the network delay amount, or to transfer the drawing command. Determine either. Here, a motion vector is used as the image feature amount used by the determination unit 185. However, other well-known feature amounts, for example, the sum of absolute values of difference values between frames, the motion compensation prediction residual error, and the like are used. An absolute value sum can also be used.
In FIG. 4, the motion vector calculation unit 201 performs, for example, D in the following Expression 1 for each block. _k Vector V to minimize _k (Dx, dy) is calculated.

Where f _{n, k} (Xi, Yj), f _{n-1, k} (Xi, Yj) represents a pixel that enters the kth block of the nth frame and a pixel that enters the kth block of the (n-1) th frame.
Next, the motion vector calculation unit 201 obtains the magnitude and direction of the motion vector for each block by the following formulas 2 and 3, and outputs these to the discrimination selection unit 202.

Where V _k Indicates the magnitude of the motion vector in the kth block, and θ _k Indicates the angle (direction) of the motion vector in the k-th block.
Next, the discrimination / selection unit 202 applies V to a plurality of consecutive blocks. _k And θ _k And in a plurality of consecutive blocks, V _k Exceeds a predetermined threshold value and θ _k If there is variation, it is determined that these blocks are moving image areas.
In addition, in a plurality of consecutive blocks, V _k Exceeds the threshold but θ _k Are substantially the same angle, it is not regarded as a moving image area, but is determined as a moving area by screen scrolling or the like. Also, V _k If the threshold does not exceed the threshold value, it is determined as a still area. Here, the moving image area is shaped so as to be a rectangular area, and the area range includes the number of pixels in the horizontal direction and the number of lines in the vertical direction of the rectangular area, and the blocks included in the area. Number and block size.
Next, the network bandwidth estimation unit 203 periodically inputs information necessary for network delay estimation from the first packet transmission / reception unit 176 of FIG. Necessary information includes, for example, the data size transmitted from the server, the time transmitted from the server, the time received by the terminal, and the like. Based on these pieces of information, for example, the network band estimated value B is periodically calculated based on the following equations 4 and 5.

D (j) indicates the data size of the jth packet transmitted from the first packet transmitting / receiving unit 176 to the mobile terminal 170, and R (j) is the jth packet received by the mobile terminal 170. Is the time of reception. These pieces of information are information that is returned from the portable terminal 170 to the first packet transmission / reception unit 176 and received by the first packet transmission / reception unit 176.
Next, the band estimation value W calculated by Expression 4 is temporally smoothed using Expression 5 below.

Here, B (n) represents a network band estimation value after smoothing at the nth time, and β is a constant in a range of 0 <β <1. The network bandwidth estimation unit 203 periodically calculates B (n) and outputs it to the discrimination selection unit 202.
Next, the discrimination / selection unit 202 obtains the ratio γ between the area of the moving image area and the area of the entire screen by the following Expression 6.
γ = Area of moving image area / area of entire screen Equation 6
Next, the discrimination / selection unit 202 inputs the network band estimation value τ from the network band estimation unit 203. If γ exceeds a predetermined threshold value Th1, it is determined that the screen is transferred, the discrimination flag F is set to 0, and the discrimination flag, the moving picture area information, and the movement area information are sent to the image encoding unit 186 in FIG. Alternatively, information on the static area is output.
When γ is less than Th1 and B (n) is less than a predetermined threshold Th2, it is determined that the command is transferred, the determination flag F is set to 1, and the determination flag F and the drawing command group are set as shown in FIG. Output to the drawing command encode 183. Further, the discrimination selection unit 202 outputs the discrimination flag F to the first packet transmission / reception unit 176.
The configuration of the image encoding unit 186 will be described with reference to FIG. In FIG. 5, a first image encoder 227 inputs an image signal of a moving image area and a determination flag, and when the determination flag is 0, that is, in the case of screen transfer, a compression code using a predetermined moving image encoder is used for the image signal. The compressed bit stream is output to the first packet transmitting / receiving unit 176 in FIG. Here, as a predetermined moving image encoder, H.264 is used. H.264 is used, but other well-known moving image codecs, for example, MPEG-4 is also available for HEVC (High Efficiency Video Coding). Further, the region information of the moving image region is output to the first packet transmitting / receiving unit 176 in FIG.
The second image encoder unit 228 inputs a captured image from the image capture unit 180, inputs a determination flag from the determination unit 185, and when the determination flag is 0, other regions (for example, a moving region or a still region) The range is input, and in the case of a still image, the image is compressed and encoded using a still image codec and output to the first packet transmitting / receiving unit 176 in FIG. Here, JPEG2000 is used as the still image codec, but other well-known codecs such as JPEG can also be used. Further, information on other areas is also output to the first packet transmission unit 176 in FIG.
On the other hand, in the case of a moving area, a bit stream obtained by compression-encoding an image before movement with a still image codec and one representative motion vector are output to the first packet transmitting / receiving unit 176 in FIG. Further, information on other areas is also output to the first packet transmitting / receiving unit 176 in FIG.
When the discrimination flag is 1, that is, when the drawing command is transferred, the drawing command encoding unit 183 inputs a drawing command group for each screen, performs lossless encoding of the drawing command group by a predetermined compression method, and results of compression encoding Is output to the first packet transmitting / receiving unit 176. Here, as the predetermined compression method, a known lossless encoding method such as a Zip compression method can be used.
Next, when audio is attached to the screen, the audio encoding unit 187 inputs an audio signal attached to the screen from the screen capture unit 180, performs compression encoding with the audio encoder, and performs the second packet transmission unit 177 in FIG. Output to. Here, MPEG-4 AAC is used as the audio encoder, but other known audio encoders can also be used.
Returning to FIG. 3, the first packet transmission / reception unit 176 inputs flag information from the determination unit. When the flag information is 0, the first packet transmission / reception unit 176 inputs the compression encoded bitstream and region information from the image encoding unit 186. When 1 is 1, a compression-encoded drawing command is input from the drawing command encoding unit 183, and flag information, area information, a compression-encoded bit stream, and a compression-encoded drawing command are stored in the payload portion of the packet. Then, a packet according to a predetermined protocol is constructed and sent to the SGSN / GGSN apparatus 190 in FIG. Here, a well-known protocol such as RTP / UDP / IP, UDP / IP, TCP / IP, or the like can be used as a predetermined protocol, but UDP / IP is used as an example. The area information can be stored in an unused area of the RTP header part or the UDP header part.
Further, the first packet transmission / reception unit 176 extracts information necessary for delay amount calculation from the response signal received from the mobile terminal 170 and outputs the information to the network bandwidth estimation unit 203 in FIG.
The second packet transmission unit 177 stores the compression-coded bit stream for the audio signal in the payload of the packet, constructs a packet according to a predetermined protocol, and transmits the packet to the SGSN / GGSN device 190. Here, a well-known protocol such as RTP / UDP / IP, UDP / IP, TCP / IP, or the like can be used as a predetermined protocol, but UDP / IP is used as an example.
The SGSN / GGSN device 190 tunnels the packet received from the server device 110 using the GTP-U protocol and transfers the packet to the RNC device 195. The RNC device 195 transmits the packet to the mobile terminal 170 wirelessly through the base station device 194.
In the present invention, client software for sending to the mobile terminal 170 an operation signal when the user operates the terminal to the server, receiving packets from the server, and decoding and displaying the compressed encoded stream is displayed. 171 is mounted. The configuration of the client software 171 is shown in FIG.
In FIG. 6, the first packet transmission / reception / selection unit 250 receives a packet, and extracts a discrimination flag, compression-encoded bitstream and area information, and compression-encoded drawing command information stored in the packet. When the determination flag is 0, that is, when screen transfer is indicated, the size of the moving image area and the compression-encoded bit stream encoded by the first image encoder are selected and extracted as moving image area information, and the first image decoder 252 Output. In addition, the other region information and the compression-encoded bit stream encoded by the other encoder in the region are output to the second image decoder 253.
The first image decoder 252 receives the moving image area information and the compression-encoded stream, decodes the compression-encoded stream, and outputs the decoded stream to the screen display unit 256. Furthermore, the moving image area information is also output to the screen display unit 256. Here, as the first image decoder, for example, H.264 is used. The H.264 decoder is used, but other well-known image decoders such as an MPEG-4 decoder can also be used. However, it goes without saying that the same type as the first image encoder 227 in the server is used.
The second image decoder 253 receives the other region information and the compression-encoded stream, decodes the compression-encoded stream for the other region, and outputs the decoded image to the screen display unit 256. Further, other area information is output to the screen display unit 256.
The screen display unit 256 receives the first area information and the image signal in the first area from the first image decoder 252, and receives the other area information and the image signal in the other area from the second image decoder 253. , The output image from the first image decoder 252 is displayed in the first area using the first area information, and the other area information is used to display the output image from the second image decoder 253 in the other area. By displaying the output image, a display screen is generated by combining the image signals of the respective regions and output.
Next, when the determination flag is 1, that is, when the drawing command is transferred, the first packet transmission / reception / selection unit 250 selects and extracts the compression-coded drawing command information and outputs it to the coder 259 with the drawing command. To do.
The drawing command decoder 259 performs, for example, Zip compression decoding and outputs a drawing command group to the screen drawing unit 260 for each screen.
The screen drawing unit 260 inputs a drawing command group for each screen, draws and generates a screen, and outputs the generated screen to the screen display unit 256.
Further, the first packet transmission / reception / selection unit 250 sends a response signal packet for the received packet to the network. Information on data size, reception time, and transmission time is described in the response signal packet.
The second packet receiving unit 251 receives the packet, extracts a compression-coded bit stream related to audio stored in the packet, and outputs the compressed bit stream to the audio decoder 255.
The audio decoder 255 receives and decodes the compression-coded stream, and outputs it in synchronization with the screen portion. Here, for example, MPEG-4 AAC can be used as the audio decoder, but other known audio decoders may be used. However, it goes without saying that the same type as the audio encoder in the server device is used.
The operation signal generation unit 257 detects an operation input by the user to the mobile terminal 170, for example, screen touch, screen scroll, icon touch, character input, and the like, generates an operation signal for each, and sends the operation signal to the packet transmission unit 258. Output.
The packet transmission unit 258 receives an operation signal, stores it in a packet according to a predetermined protocol, and transmits it to the network. Here, TCP / IP, UDP / IP, etc. can be used as a predetermined protocol.
According to the present embodiment, when a thin client is used via a network, an image feature amount is obtained for a screen generated on the server side, and at least one of the feature amount and a network bandwidth estimation value or a network delay amount is obtained. Based on one, it is determined whether to transfer the screen or drawing command, and these are switched and transferred, so it is realized by balancing the amount of data, the amount of network delay, and the load on the terminal As a result, it is possible to prevent a significant increase in data acid, a significant delay in response time, and a significant increase in processing load at the terminal.
While the present invention has been described with reference to the embodiment, the present invention is not limited to this.
There may be three or more types of screen areas determined by the determination / selection unit 202. In addition, a feature quantity other than a motion vector can be used as an image feature quantity used for region determination, or a plurality of types of feature quantities can be used in combination.
Other known methods can be used as the bandwidth estimation method in the network bandwidth estimation unit 203. Further, instead of band estimation, the delay amount can be estimated, and the delay selection unit 202 can use the delay amount estimation value to determine whether the screen transfer or the drawing command transfer.
Other known methods can be used as the determination method in the determination / selection unit 202.
In FIG. 2, the mobile network 150 can be a mobile LTE / EPC network, or can be a WiMax network or a WiFi network. Further, it can be a fixed network, an NGN network, or an Internet network. However, in these cases, connection is made from a fixed terminal or a PC instead of a portable terminal.
In FIG. 2, the server device is arranged in the cloud network, but can be arranged in the Internet network. In addition, when a thin client server is installed in a company, the server device can be arranged in the company network. As another configuration, when the telecommunications carrier itself places a thin client server, the server device 110 can be arranged on the mobile network 150, a fixed network, or an NGN network.
A part or all of the above embodiments may be described as in the following supplementary notes, but is not limited thereto.
(Appendix 1)
A transmission-side device that transmits a moving image for each screen via a data communication network, and a reception-side device that receives the screen via the data communication network,
The transmitting device is:
Means for dividing the screen into a plurality of blocks;
Means for selecting at least one of the plurality of blocks, either image data obtained by encoding the block, or a drawing command for drawing the block;
Means for encoding the block into image data;
Means for determining a drawing command for drawing the block from the block; and
Means for transmitting the image data and the drawing command to the receiving side device via the data communication network;
The receiving side device
Means for discriminating the image data and drawing command received from the transmission side device via the data communication network;
Means for decoding the image data and outputting as an image signal constituting a part of the screen; and
Means for outputting an image signal of an image obtained by drawing a part of the screen in accordance with the drawing command;
A video transmission system characterized by the above.
(Appendix 2)
The transmitting device operates as the server of a thin client system including a server that executes a virtual client and provides a screen of a virtual client to a client terminal, and a client terminal that displays a screen provided from the server. ,
The receiving device operates as the client terminal;
The data communication network includes a mobile communication network
The moving image transmission system according to supplementary note 1, wherein:
(Appendix 3)
The system according to any one of Supplementary Note 1 and Supplementary Note 2, wherein the means for selecting one of the image data and the drawing command performs selection according to one or more image feature amounts obtained from the block. .
(Appendix 4)
The system according to any one of appendix 1 to appendix 3, wherein the means for selecting one of the image data and the drawing command performs selection according to a communication status of the data communication network.
(Appendix 5)
The transmission side device includes a plurality of image encoders having different encoding methods as means for encoding the image data,
The receiving-side apparatus includes a plurality of image decoders corresponding to the encoding methods of the plurality of image encoders as means for decoding the image data.
The system according to any one of appendix 1 to appendix 4, wherein
(Appendix 6)
The transmission-side apparatus includes, as means for encoding the image data, a first image encoder for encoding a moving image area and a second image encoder for encoding a non-moving area,
The receiving-side apparatus includes, as means for decoding the image data, a first image encoder for decoding a moving image area and a second image encoder for decoding a non-moving area. The system according to appendix 5, which is characterized.
(Appendix 7)
A way to divide one screen of a video into multiple blocks,
Means for selecting at least one of the plurality of blocks, either image data obtained by encoding the block, or a drawing command for drawing the block;
Means for encoding the block into image data;
Means for determining a drawing command for drawing the block from the block; and
A data communication apparatus comprising means for transmitting the image data and the drawing command to another data communication apparatus via a data communication network.
(Appendix 8)
The computer operates as the server of a thin client system including a server that executes a virtual client and provides a screen of the virtual client to the client terminal, and a client terminal that displays the screen provided from the server,
The other data communication device operates as the client terminal,
The data communication network includes a mobile communication network
The data communication device according to appendix 7, wherein
(Appendix 9)
The data according to any one of appendix 7 and appendix 8, wherein the means for selecting one of the image data and the drawing command performs selection according to one or more image feature amounts obtained from the block. Communication device.
(Appendix 10)
10. The data communication apparatus according to any one of appendix 7 to appendix 9, wherein the means for selecting one of the image data and the drawing command performs selection according to a communication status of the data communication network.
(Appendix 11)
11. The data communication apparatus according to any one of appendix 7 to appendix 10, wherein the means for encoding the image data functions as a plurality of image encoders having different encoding methods.
(Appendix 12)
Additional means 11 for encoding the image data functions as a first image encoder for encoding a moving image area and a second image encoder for encoding a non-moving area. The data communication device described.
(Appendix 13)
Means corresponding to one of a plurality of blocks formed by dividing one screen of a moving image, and means for discriminating image data or a drawing command received from the other data communication device via a data communication network;
Means for decoding the image data and outputting as an image signal constituting a part of the screen; and
Means for outputting an image signal of an image obtained by drawing a part of the screen in accordance with the drawing command;
A data communication device.
(Appendix 14)
The other data communication device is a server of a thin client system including a server that executes a virtual client and provides a screen of the virtual client to the client terminal, and a client terminal that displays the screen provided from the server. Work,
The data communication device operates as the client terminal;
The data communication network includes a mobile communication network
14. The data communication device according to appendix 13, wherein
(Appendix 15)
15. The data communication apparatus according to any one of supplementary note 13 and supplementary note 14, comprising a plurality of image decoders corresponding to a plurality of image encoder encoding methods as means for decoding the image data.
(Appendix 16)
(Supplementary note 15) The means for decoding the image data includes a first image encoder for decoding a moving image area and a second image encoder for decoding a non-moving area. The data communication device described.
(Appendix 17)
A way to divide one screen of a video into multiple blocks,
Means for selecting at least one of the plurality of blocks, either image data obtained by encoding the block, or a drawing command for drawing the block;
Means for encoding the block into image data;
Means for determining a drawing command for drawing the block from the block; and
A program for causing a computer to function as means for transmitting the image data and the drawing command to another data communication apparatus via a data communication network.
(Appendix 18)
The computer operates as the server of a thin client system including a server that executes a virtual client and provides a screen of the virtual client to the client terminal, and a client terminal that displays the screen provided from the server,
The other data communication device operates as the client terminal,
The data communication network includes a mobile communication network
The program according to appendix 17, characterized by:
(Appendix 19)
The program according to any one of appendix 17 and appendix 18, wherein the means for selecting one of the image data and the drawing command performs selection according to one or more image feature amounts obtained from the block. .
(Appendix 20)
The program according to any one of appendix 17 to appendix 19, wherein the means for selecting one of the image data and the drawing command performs selection according to a communication status of the data communication network.
(Appendix 21)
The program according to any one of appendix 17 to appendix 20, wherein the means for encoding the image data functions as a plurality of image encoders having different encoding methods.
(Appendix 22)
The appendix 21 is characterized in that the means for encoding the image data functions as a first image encoder for encoding a moving image area and a second image encoder for encoding a non-moving area. The listed program.
(Appendix 23)
Means corresponding to one of a plurality of blocks formed by dividing one screen of a moving image, and means for discriminating image data or a drawing command received from the other data communication device via a data communication network;
Means for decoding the image data and outputting as an image signal constituting a part of the screen; and
A program for causing a computer to function as means for outputting an image signal of an image obtained by drawing a part of the screen in accordance with the drawing command.
(Appendix 24)
The other data communication device is a server of a thin client system including a server that executes a virtual client and provides a screen of the virtual client to the client terminal, and a client terminal that displays the screen provided from the server. Work,
The data communication device operates as the client terminal;
The data communication network includes a mobile communication network
The program according to appendix 23, which is characterized by the above.
(Appendix 25)
25. The program according to any one of Supplementary Note 23 and Supplementary Note 24, wherein the means for decoding the image data functions as a plurality of image decoders corresponding to a plurality of image encoder encoding methods.
(Appendix 26)
Appendix 25 wherein the means for decoding the image data functions as a first image encoder for decoding a moving image area and a second image encoder for decoding a non-moving area The listed program.
(Appendix 27)
Dividing one screen of the video into a plurality of blocks at the transmission side device;
In the transmitting device, for one of the plurality of blocks, selecting one of encoding of the block into image data and determination of a drawing command for drawing the block;
Performing any one of the above in response to the selection at the transmitting device;
Transmitting either the image data or the drawing command from the transmitting device to the receiving device via the data communication network; and
A step of generating an image signal constituting a part of the screen by performing any one of decoding of the image data and drawing of an image according to the drawing command in the receiving side device.
A moving picture transmission method comprising:
(Appendix 28)
The transmitting device operates as the server of a thin client system including a server that executes a virtual client and provides a screen of a virtual client to a client terminal, and a client terminal that displays a screen provided from the server. ,
The receiving device operates as the client terminal;
The data communication network includes a mobile communication network
The method according to appendix 27, wherein:
(Appendix 29)
The step of selecting either one of the encoding to the image data and the determination of the drawing command is performed according to one or more image feature amounts obtained from the block. The method according to any one.
(Appendix 30)
30. The method according to any one of appendixes 27 to 29, wherein the step of selecting one of the encoding to the image data and the determination of the drawing command is performed according to a communication status of the data communication network. the method of.
(Appendix 31)
The image data is encoded using one image encoder selected from a plurality of image encoders having different encoding methods,
The decoding of the image data is performed using an image decoder corresponding to the encoding method of the selected one image encoder.
31. The method according to any one of appendix 27 to appendix 30, wherein
(Appendix 32)
The image data is encoded using either a first image encoder for encoding a moving image region and a second image encoder for encoding a non-moving region,
The decoding of the image data is performed using either a first image encoder for decoding a moving image area or a second image encoder for decoding a non-moving area. 31. The method according to 31.
This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2012-226654 for which it applied on October 12, 2012, and takes in those the indications of all here.

Claims

A transmission-side device that transmits a moving image for each screen via a data communication network, and a reception-side device that receives the screen via the data communication network,
The transmitting device is:
Means for dividing the screen into a plurality of blocks;
Means for selecting at least one of the plurality of blocks, either image data obtained by encoding the block, or a drawing command for drawing the block;
Means for encoding the block into image data;
Means for determining a drawing command for drawing the block from the block; and
Means for transmitting the image data and the drawing command to the receiving side device via the data communication network;
The receiving side device
Means for discriminating the image data and drawing command received from the transmission side device via the data communication network;
Means for decoding the image data and outputting as an image signal constituting a part of the screen; and
A moving picture transmission system comprising: means for outputting an image signal of an image obtained by drawing a part of the screen in accordance with the drawing command.
The transmitting device operates as the server of a thin client system including a server that executes a virtual client and provides a screen of a virtual client to a client terminal, and a client terminal that displays a screen provided from the server. ,
The receiving device operates as the client terminal;
The moving image transmission system according to claim 1, wherein the data communication network includes a mobile communication network.
The means for selecting any one of the image data and the drawing command performs selection according to one or more image feature amounts obtained from the block. Video transmission system.
4. The moving picture transmission system according to claim 1, wherein the means for selecting one of the image data and the drawing command performs selection according to a communication status of the data communication network. .
The transmission side device includes a plurality of image encoders having different encoding methods as means for encoding the image data,
The receiving apparatus includes a plurality of image decoders corresponding to encoding methods of the plurality of image encoders as means for decoding the image data. The video transmission system described.
The transmission-side apparatus includes, as means for encoding the image data, a first image encoder for encoding a moving image area and a second image encoder for encoding a non-moving area,
The receiving-side apparatus includes, as means for decoding the image data, a first image encoder for decoding a moving image area and a second image encoder for decoding a non-moving area. 6. The moving image transmission system according to claim 5, wherein
A way to divide one screen of a video into multiple blocks,
Means for selecting at least one of the plurality of blocks, either image data obtained by encoding the block, or a drawing command for drawing the block;
Means for encoding the block into image data;
Means for determining a drawing command for drawing the block from the block; and
A data communication apparatus comprising means for transmitting the image data and the drawing command to another data communication apparatus via a data communication network.
Means corresponding to one of a plurality of blocks formed by dividing one screen of a moving image, and means for discriminating image data or a drawing command received from the other data communication device via a data communication network;
Means for decoding the image data and outputting as an image signal constituting a part of the screen; and
A data communication apparatus comprising: means for outputting an image signal of an image obtained by drawing a part of the screen in accordance with the drawing command.
A way to divide one screen of a video into multiple blocks,
Means for selecting at least one of the plurality of blocks, either image data obtained by encoding the block, or a drawing command for drawing the block;
Means for encoding the block into image data;
Means for determining a drawing command for drawing the block from the block; and
A program for causing a computer to function as means for transmitting the image data and the drawing command to another data communication apparatus via a data communication network.
Dividing one screen of the video into a plurality of blocks at the transmission side device;
In the transmitting device, for one of the plurality of blocks, selecting one of encoding of the block into image data and determination of a drawing command for drawing the block;
Performing any one of the above in response to the selection at the transmitting device;
Transmitting either the image data or the drawing command from the transmitting device to the receiving device via the data communication network; and
Including a step of generating an image signal constituting a part of the screen by performing any one of decoding of the image data and drawing of an image according to the drawing command in the receiving-side apparatus. A video transmission method characterized by the above.