JP2005151298A - Image transmitting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image transmitting device in which all image recipients can view the same image with high quality regardless of the difference of image reception environments even in an environment where bandwidths of communications paths that the image recipients have, or throughputs of display apparatuses are variously different. <P>SOLUTION: An encoding and storage means hierarchically encodes object images, generates and stores fundamental hierarchy and extension hierarchy encoded streams of the object images. In the extension hierarchy, bit plane encoding is performed. In the case of encoding, a rate distortion property is generated and held, the rate distortion property indicating the relationship of a code amount and an encoding distortion when code words of the extension hierarchy encoded streams for the object images are sequentially decoded. A transmission means distributes the code amount for each object image on the basis of the rate distortion property and the bandwidth of a communication path 13, cuts and sends the extension hierarchy encoded streams for each object according to the distribution. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image transmission apparatus, and in particular, even in an environment where the bandwidth of a communication path and the processing capability of a display device are variously different, all image recipients can obtain the same image with high quality regardless of the difference in the receiving environment. The present invention relates to an image transmission apparatus that can be viewed.

  In recent years, digital image distribution has become easier due to improved computer performance and wider network bandwidth. On the other hand, due to the diversification of networks, the bandwidth of the communication path possessed by the image receiver and the processing capability of the display device are different. From the viewpoint of broadcasting, it is important that all image receivers can view the same image regardless of differences in the image receiving environment even when different image receiving environments exist.

  Hierarchical coding technology exists as a method for solving this problem. In the hierarchical encoding technique, an encoded stream divided into two layers, a basic layer and an extended layer, is created. The basic layer includes minimum information for reproducing an image, and the extended layer includes information added to the basic layer to enhance the quality of a reproduced image. Hierarchical coding technology enables a scalable transmission layer to be selected according to the bandwidth of a communication channel possessed by an image receiver and the processing capability of a display device, and an image can be reproduced with two levels of quality.

  As one of the conventional typical hierarchical encoding techniques, there can be mentioned FGS (Fine Granularity Scalability) encoding technique proposed in Non-Patent Document 1 below. The FGS encoding technique is a hierarchical encoding technique with particularly high scalability (degree of freedom in decoding). This high scalability is realized by a bit plane encoding process. That is, in the enhancement layer of the FGS encoding technique, information for improving the image quality is finely encoded by the bit plane encoding process, and a code word of a relatively small unit is created. Codewords created by the bit-plane coding process are stored in the enhancement layer coded stream in descending order of contribution to the quality of the reproduced image.

Therefore, the code amount of the enhancement layer encoded stream can be changed to flexibly control the code amount according to the bandwidth of the communication channel of the image receiver, and at the same time, a high quality image can be reproduced. . In addition, even when an enhancement layer encoded stream is lost during transmission due to communication path congestion, it is possible to reproduce an image having a quality corresponding to the code amount of the received encoded stream.
Weiping Li, “Overviw of Fine Granularity Scalability in MPEG-4 Video Standard”, IEEE Transaction on circuit and systems for video technology, Vol.11, No.3, Mar.2001

  However, the conventional hierarchical coding technique has a low efficiency of information compression by coding due to the effect of division loss due to hierarchization (decrease in motion compensation accuracy due to hierarchization), and compared with a coding method that does not perform hierarchization. Thus, there is a problem that the quality of the reproduced image is significantly lower. In other words, the basic principle of image coding technology is to improve the quality of a reproduced image with a limited amount of information by efficiently removing redundant signals in the image. There is a problem that the process of removing the signal cannot be performed, the encoding efficiency is lowered, and the quality of the reproduced image is lowered.

  For example, in the image coding technique, it is possible to efficiently reduce redundant signals by performing motion compensation prediction processing of the image between the previous and subsequent frames, but in the FGS coding, the motion compensation prediction processing is applied only to the base layer, Since only the simple inter-frame prediction process is applied to the extension layer, redundant signals cannot be removed efficiently. Such a decrease in encoding efficiency lowers the quality of the entire reproduced image of the image receiver, and further reduces the subjective quality and semantic content that the receiver receives from the image.

  The object of the present invention is to solve the above-mentioned problems, and even in an environment where the bandwidth of the communication channel possessed by the image receiver and the processing capability of the display device are different, all the image receivers are not related to the difference in the image receiving environment. An object of the present invention is to provide an image transmission apparatus capable of viewing the same image with high quality.

  In order to solve the above-described problems, the present invention provides an image transmission apparatus that transmits an image by performing hierarchical encoding processing, and divides the original image into a plurality of object images, and each object image is divided into two layers: a basic layer and an extended layer. In the base layer, encoding processing including quantization processing is performed to generate a base layer encoded stream, and in the enhancement layer, a residual image that is a difference between the original image and the decoded image of the base layer is bit-plane encoded. To generate an enhancement layer coded stream, accumulate the generated base layer coded stream and enhancement layer coded stream, and code the code of the enhancement layer coded stream sequentially decoded for each object image Encoding storage means for generating and holding a rate distortion characteristic indicating the relationship between the amount and the encoding distortion, the rate distortion characteristic and the bandwidth of the communication channel, Based on, allocated code amount for each object image, there is a first feature in that a transmission means for transmitting to cut enhancement layer encoded stream of each object image according 該配 min.

  Further, the present invention has a second feature in that the transmission means weights each object image when allocating a code amount for each object image.

  According to the features of the present invention, it is possible to control the maximum image quality of the entire frame in consideration of the rate distortion characteristics and the bandwidth of the communication channel, so that the bandwidth of the communication channel and the processing capability of the display device vary. Even in different environments, all image recipients can view the same image with high quality regardless of the difference in the receiving environment.

  Also, when allocating the code amount for each object image, for example, the foreground object image is relatively high quality and the background object image is relatively low by weighting each object image according to visual complexity, priority, etc. The subjective quality of the entire frame image can be enhanced by encoding with quality.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a basic configuration of an image transmission apparatus according to the present invention. Hereinafter, a case where an image is distributed to an image recipient will be described as an example.

  First, an input image is input to the encoding / storage device 11. The input image is a plurality of object images o (1), o (2),... In which the part other than the object is set to “0” level, for example, which divides the image area of the object from the frame image Obtained by.

  A known technique can be used to divide the object image from the frame image. For example, the object extraction described in Japanese Patent Laid-Open No. 2003-44860 “Video Object Tracking Device”, Japanese Patent Laid-Open No. 2002-358526 “Video Object Detection / Tracking Device”, and Japanese Patent Laid-Open No. 2001-236512 “Moving Object Extraction Device” Technology can be used.

The encoding storage device 11 performs hierarchical encoding processing on each object image o (1), o (2),..., And for each object image, encoded streams a (1), a in the base layer and the extended layer. (2), ... and rate distortion characteristics b (1), b (2), ... are generated and stored. Details of the rate distortion characteristics will be described later. Note that bit-plane encoding is performed in the enhancement layer.
The transmission apparatus 12 reduces the code amount of the encoded streams a (1), a (2),... According to the rate distortion characteristics b (1), b (2),. The encoded streams a ′ (1), a ′ (2),... Are generated and sent to the communication path 13. The code amount of the encoded streams a (1), a (2),... Is reduced by cutting the enhancement layer encoded stream that has been bit-plane encoded. The extension layer encoded streams a ′ (1), a ′ (2),... Are transmitted based on, for example, a request from the image receiver or time, and the image is distributed to the image receiver. Is done.

  The transmitted encoded streams a ′ (1), a ′ (2),... Have a code amount that can be transmitted within the bandwidth of the communication path 13 as a whole, and maximize the quality of the reproduced image. It is assumed that the code amount allocated so as to be increased is set. For this purpose, the communication channel bandwidth c measured prior to image transmission is input to the transmission device 12.

  The reproduction device 14 on the image receiver side decodes the encoded streams a ′ (1), a ′ (2),... Received from the communication path 13 and reproduces the reproduced object images o ′ (1), o ′ ( 2) Play ... As a result, the image receiver can obtain the reproduction object images o ′ (1), o ′ (2),... With the maximum quality that can be received by the communication channel 13 that the image receiver has. A frame image can be viewed by integrating o ′ (1), o ′ (2),.

  FIG. 2 is a diagram conceptually illustrating a transmission form when there are two object images. First, the frame image is divided into object images o (1) and o (2). The object image o (1) is a foreground image extracted from a frame image, for example, and the object image o (2) is a background image thereof.

  Next, the object images o (1) and o (2) are hierarchically encoded to generate encoded streams a (1) and a (2). The encoded streams a (1) and a (2) include a base layer encoded stream and an enhancement layer encoded stream. In general, the amount of code decreases as the image moves less.

  The code amount of the encoded streams a (1) and a (2) of the object images o (1) and o (2) is determined according to the rate distortion characteristics b (1) and b (2) and the channel bandwidth c. The unit is cut according to the communication channel bandwidth to obtain encoded streams a ′ (1) and a ′ (2), which are transmitted to the communication channel.

  On the receiving side, the encoded streams a ′ (1) and a ′ (2) are received, the object images o ′ (1) and o ′ (2) are reproduced, and they are integrated to reconstruct a frame image. .

  A specific example of each apparatus in FIG. 1 will be described below by taking as an example the case of using an NPEG-4 based FGS encoding technique as a hierarchical encoding technique.

Encoding storage device 11
FIG. 3 is a block diagram illustrating a configuration example of the encoding / storage device 11. FIG. 3 shows a configuration for processing one of the object images o (1), o (2),... Separated from the frame image. The plurality of object images o (1), o (2),... Are processed using a plurality of such configurations in parallel or one in time division. In the following description, it is assumed that the configuration of FIG. 3 processes the object image o (1).

  The input object image o (1) is input to the subtracters 31 and 32 and the motion detector 33. A base layer encoding process is performed at a portion subsequent to the subtractor 31, and an extension layer encoding process is performed at a portion subsequent to the subtractor 32.

  In the base layer encoding process, first, the prediction image obtained by the motion compensation prediction unit 34 is subtracted from the input object image o (1) by the subtractor 31 to generate a prediction error image. This prediction error image is converted into a DCT coefficient by a DCT (orthogonal transform) unit 35, for example, in block units of 8 pixels in the vertical direction and 8 pixels in the horizontal direction.

  The quantization unit 36 quantizes the DCT coefficient according to the quantization parameter. The quantized DCT coefficients are variable length encoded by a variable length encoding (VLC) unit 37 and multiplexed with motion vector information MV to generate a base layer encoded stream d (1).

  Also, the quantized DCT coefficients from the quantization unit 36 are inversely quantized by the inverse quantization unit 38, reproduced as a prediction error image by the inverse DCT unit 39, and predicted from the motion compensation prediction unit 34 by the adder 40. It is added to the image. The locally decoded image obtained by the adder 40 is stored in the frame memory 41.

  The motion detection unit 33 detects the motion of the object image o (1) based on the sequentially input object image o (1) and the local decoded image from the frame memory 41, and outputs motion vector information MV. The motion vector information MV is given to the motion compensation prediction unit 34 and the variable length coding unit 37. The motion of the object image is detected, for example, in units of macroblocks of 16 pixels vertically and 16 pixels horizontally. The motion compensation prediction unit 34 generates a prediction image from the locally decoded image from the frame memory 41 and the motion vector information MV from the motion detection unit 33 and outputs the prediction image to the subtracter 31 and the adder 40.

  What is to be encoded in the enhancement layer encoding process is an output of the subtracter 32, that is, a residual image that is a difference between the input object image o (1) and the locally decoded image stored in the frame memory 41. . This residual image is converted into DCT coefficients by the DCT unit 42. The DCT coefficients thus generated are encoded in bit plane units by the bit plane encoding unit 43, and further variable length encoded by the variable length encoding unit 44 to generate an enhancement layer encoded stream e (1). The At the time of this encoding, the rate distortion characteristic f (1) is generated by the bit plane encoding unit 43.

  FIG. 4 is an explanatory diagram of the encoding process procedure of the enhancement layer of the FGS encoding technique. In the extension layer of the FGS encoding technique, DCT is performed on the difference image in units of blocks of 8 pixels in the vertical direction and 8 pixels in the horizontal direction to obtain DCT coefficients. Next, the DCT coefficients of each block are zigzag scanned (FIG. 4A), and the DCT coefficients are rearranged into a primary series (FIG. 4B), and further converted into a bit plane representation (FIG. 4C). After that, the binary information symbol of each bit plane is encoded by binary zero-run length encoding processing (RUN, EOP) to create a code word ((d) in the figure).

  According to the above encoding processing procedure, in the order of one pass from the one that contributes to the quality of the object image when reproduced (in FIG. 4D, (1,1) (0,0) (0, Each codeword is stored in the encoded stream in the order of 0) (1,0) (0,1) (0,0) (1,0) (2,0). As described above, in the hierarchical encoding technique in which each codeword is stored in the encoded stream in the order of one pass from the one that contributes to the quality of the object image when played back, the code amount of the codeword to be decoded R and coding distortion D can be measured, and therefore, the coding distortion D with respect to the code amount can be known in codeword units.

  Here, the coding distortion D indicates a difference between the reproduction object image and the original object image when the decoding process is terminated at the code word, and the coding distortion D decreases as more code words are decoded. . For example, in the example of the FGS encoding technique, the encoding distortion D can be defined by a mean squared error (MSE) from the original image when the objective image quality is taken into account. It is also possible to define the weighted mean square error (weighted MSE) in consideration of quality. Depending on which definition is used to measure the coding distortion D on a scale, it is determined whether the quality of the reproduced object image is objectively improved or subjectively improved.

In the present invention, in order to perform image transmission so as to maximize the quality of a reproduced object image, a rate distortion characteristic which is a relationship between a code amount R of a codeword to be decoded and a coding distortion D is set for each object image. Measure for o. That is, the total code amount in the case of aborted in j _o th codeword of the object image o the R _o ^jo, when aborted by j _o th codeword of the object image o the reproduced object image and the original object image When the difference is D _o ^jo , the code amount R _o ^jo and the coding distortion D _o ^jo are measured every time one codeword is created in the hierarchical coding process of the object image o. As shown in the graph, the horizontal axis is R _o ^jo and the vertical axis is D _o ^jo . This is performed for each object image o (1), o (2),..., And rate distortion characteristics f (1), f for each object image o (1), o (2),. (2), ...

The graphs of the rate distortion characteristics f (1), f (2),... ^Are monotonically decreasing characteristics in which the encoding distortion D _o ^jo decreases as the code amount R _o ^jo increases. Like the object image o (1) in FIG. 5, an image with a small code amount R _o ^{jo and a} small coding distortion D _o ^jo is generally an image with little motion.

  As described above, the encoding storage device 11 performs the base layer encoded streams d (1), d (2),... For each object image o (1), o (2),. .. Are generated, stored and held, and further, rate distortion characteristics f () for each object image o (1), o (2),. 1), f (2),... Are generated and held.

Transmission device 12
FIG. 6 is a block diagram illustrating a configuration example of the transmission apparatus 12. The transmission device 12 includes base layer and enhancement layer encoded streams d (1), d (2),..., E (1), e (2),. , Rate distortion characteristics f (1), f (2),..., And bandwidth c of the communication path 13 between the image receivers are input.

  The code amount distribution processing unit 61 includes rate distortion characteristics f (1), f (2),..., The bandwidth c of the communication path 13, and the base layer encoded stream measured by the base layer code amount measuring unit 62. The code amount to be assigned to each object image o (1), o (2),... is determined using the total code amount B of d (1), d (2),.

  The stream cutters 63 (1), 63 (2),... Are enhancement layer encoded streams e (1) of the object images o (1), o (2),. , e (2),..., and an extended layer coded stream e ′ (1), e ′ (2),. These extension layer encoded streams e ′ (1), e ′ (2),... Are sent to the communication path 13 together with the base layer encoded streams d (1), d (2),. It is transmitted to the image recipient.

  The code amount distribution processing unit 61 determines the code amount distribution so as to maximize the quality of the reproduced image within the range of the code amount that can be transmitted through the communication path 13 with the image receiver. Specifically, the rate distortion characteristics f (1), f (2),..., The bandwidth c of the communication path 13, and the encoded stream d (1), d (2),. , The code amount to be assigned to each object image o (1), o (2),... Is calculated by, for example, the following algorithm.

  If the code amount that can be transmitted determined from the bandwidth c of the communication path 13 with the image receiver is C, and the code amount of the base layer encoded stream d (1), d (2),. The set code amount R that can be allocated to the encoded streams e ′ (1), e ′ (2),... Of the entire enhancement layer is represented by R = C−B.

In order to maximize the image quality of the reproduced object images o ′ (1), o ′ (2),..., The rate distortion characteristics of the object images o (1), o (2),. It is necessary to minimize the next evaluation function F using f (1), f (2),.

F = Σ _o (k _o D _o ^jo + λR _o ^jo )

Here, λ is a constant, and the combination of the code amount R _o ^jo and the encoding distortion D _o ^jo that minimizes the evaluation function F of the above equation with respect to a certain constant λ is uniquely determined. K _o is a weighting factor determined by the visual complexity, priority, etc. of each object image o (1), o (2),. For example, a combination of the code amount R _o ^jo and the coding distortion D _o ^jo is determined so that a larger code amount is assigned as k _o is larger, and a smaller code amount is assigned as k _o is smaller. Thus, for example, the subjective quality of the entire frame image can be improved by encoding the foreground object image with a relatively high quality and the background object image with a relatively low quality.

The constant λ corresponds to the tangential slope dD _o / dR _o in the rate distortion characteristic curve. For a certain value of λ, minimizing the evaluation function F has the same slope λ for the coding distortion characteristic curves for all object images o (1), o (2),. Equivalent to finding the tangent. When the tangent line having the same slope λ is obtained with respect to the coding distortion characteristic curve for all object images o (1), o (2),..., It is truncated at the j _o (λ) -th code word at the contact point. In this case, the code amount R _o ^{jo (λ)} and the coding distortion D _o ^{jo (λ)} are a combination that minimizes the evaluation function F.

When the code amount R _o ^{jo (λ)} and the coding distortion D _o ^{jo (λ)} that minimize the evaluation function F with respect to a certain value of λ are obtained in this way, each object image o (1 ), o (2),..., and the constant λ so that the following restrictions are imposed so that the total code amount allocated to the entire extended hierarchy of the frame screen does not exceed the set code amount R Determine the value of.

When the total amount of code amount R _o ^{jo (λ)} to be assigned to the enhancement layer encoded stream of each object image o (1), o (2) ^,.

R (λ) = Σ _o R _o ^{jo (λ)} <(set code amount R that can be allocated to the encoded streams e ′ (1), e ′ (2),... Of the entire enhancement layer)

And the constant λ is determined so that the code amount of the encoded stream e ′ (1), e ′ (2),... Of the entire enhancement layer is as close as possible to the set code amount R. When the enhancement layer encoded stream e (1), e (2),... Is terminated with the j _o (λ) -th codeword corresponding to the constant λ at this time, the reproduction object image o ′ (1), o '(2), ... The overall image quality can be maximized.

FIG. 7 is an explanatory diagram showing the concept of the code amount distribution process when there are two object images. A tangent having the same slope λ is obtained with respect to the coding distortion characteristics f (1) and f (2) for the object images o (1) and o (2), and the code amount R ₁ ^{j1 (λ)} R ₂ ^{j2 (λ)} is obtained. The constant λ is changed so that (R ₁ ^{j1 (λ)} + R ₂ ^{j2 (λ)} ) is less than or equal to the set code amount R, and the code amounts R ₁ ^{j1 (λ)} and R ₂ ^{j2 at that time Find (λ)} . When the encoded stream is transmitted to the communication path, the encoded stream of the enhancement layer of the object image o (1) is cut off at the j1 (λ) -th codeword, and the encoded stream of the enhancement layer of the object image o (2) Is cut off at the j2 (λ) -th codeword.

Playback device 14
FIG. 8 is a block diagram illustrating a configuration example of the playback device 14. FIG. 8 shows a configuration for reproducing one of the object images o ′ (1), o ′ (2),. A plurality of object images o ′ (1), o ′ (2),... Are reproduced in parallel by providing a plurality of such configurations, or using one configuration in time division. In the following description, it is assumed that the configuration of FIG. 8 reproduces the object image o ′ (1).

  The base layer encoded stream d (1) is variable length decoded by the variable length decoding (VLD) unit 81, and the quantized DCT coefficient is output. At the same time, motion vector information MV is output. The quantized DCT coefficient is inversely quantized by the inverse quantization unit 82 and is reproduced as a prediction error image by the inverse DCT unit 83. The adder 84 adds the prediction error signal and the prediction image from the motion compensation prediction unit 85, and outputs a base layer object image. The basic layer object image is output to the adder 90 and is stored in the frame memory 86. The decoded image stored in the frame memory 86 is given to the motion compensation prediction unit 85.

  The enhancement layer encoded stream e ′ (1) is subjected to variable length decoding by the variable length decoding unit 87, and is decoded into residual image DCT coefficients by the bit plane decoding unit 88. This DCT coefficient is converted into a residual image by the inverse DCT unit 89. The adder 90 adds the residual image and the base layer object image, and outputs a reproduction object image o ′ (1).

  As is clear from the above description, according to the present invention, it is possible to perform control for maximizing the image quality of the entire frame in consideration of the rate distortion characteristics and the channel bandwidth, so that a network in which different image receiving environments are mixed is used. It can be effectively applied to the above image distribution system.

It is a block diagram which shows the basic composition of the image transmission apparatus which concerns on this invention. It is a figure which shows notionally the transmission form by this invention. It is a block diagram which shows the structural example of an encoding storage apparatus. It is explanatory drawing of the encoding process procedure of the extended hierarchy of FGS encoding technique. It is explanatory drawing of a rate distortion characteristic. It is a block diagram which shows the structural example of a transmission apparatus. It is explanatory drawing which shows the concept of code amount distribution processing. It is a block diagram which shows the structural example of the reproducing | regenerating apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Coding storage apparatus, 12 ... Transmission apparatus, 13 ... Communication path, 14 ... Reproduction | regeneration apparatus, 31, 32 ... Subtractor, 33 ... Motion detection part, 34, 85 ... motion compensation unit, 35, 42 ... DCT (orthogonal transform) unit, 36 ... quantization unit, 37, 44 ... variable length coding (VLC) unit, 38, 82 ... reverse Quantization unit, 39, 83, 89 ... inverse DCT unit, 40, 84, 90 ... adder, 41, 86 ... frame memory, 43 ... bit plane encoding unit, 61 ... Code amount distribution processing unit, 62 ... base layer code amount measurement unit, 63 (1), 63 (2) ... stream cutter, 81, 87 ... variable length decoding unit, 88 ... bit plane decoding Part

Claims (2)

In an image transmission apparatus that transmits images after hierarchical encoding processing,
The original image is divided into a plurality of object images, each object image is divided into two layers, a basic layer and an extended layer, and a basic layer encoded stream is generated by performing an encoding process including a quantization process in the basic layer. In the layer, a residual image, which is the difference between the original image and the decoded image of the base layer, is subjected to bit-plane encoding processing to generate an extended layer encoded stream, and the generated base layer encoded stream and the extended layer encoded stream are stored. And encoding and storing means for generating and holding rate distortion characteristics indicating the relationship between the code amount and the encoding distortion when the codeword of the enhancement layer encoded stream is sequentially decoded for each object image,
Transmission means for allocating a code amount for each object image on the basis of the rate distortion characteristic and the bandwidth of the communication channel, and cutting and transmitting an enhancement layer encoded stream of each object image according to the distribution An image transmission device. The image transmission apparatus according to claim 1, wherein the transmission unit weights each object image when allocating a code amount for each object image.

Images