JP5484377B2 - Decoding device and decoding method - Google Patents

Description

  The present invention relates to a decoding device and a decoding method. In particular, the present invention relates to a decoding apparatus and a decoding method that perform super-resolvable region determination for distributed encoding.

  With the increasing demand for multimedia content, research on video compression has been actively conducted. A new method called Distributed Video Coding (DVC) has been proposed and attracted attention for video compression methods such as H.264 / AVC which are now widely used.

  In the conventional video compression method, high-load processing is performed on the encoding side, whereas in DVC, high-load processing is performed on the decoding side. Taking advantage of this feature, it is expected that even a terminal having a low processing capability such as a mobile phone can distribute video with the same coding efficiency as that of the conventional video compression method. However, DVC has a short history of research, and has not yet achieved encoding efficiency comparable to that of conventional video compression methods.

  First, distributed video coding (DVC) and super resolution (SR) will be described.

(1) Distributed video coding (DVC)
FIG. 1 shows a block diagram of a general DVC encoder / decoder (see Non-Patent Document 1). The encoding side performs intra encoding by DCT conversion or Wavelet conversion every few frames. This encoded frame is called a Key frame. Other frames called Wyner-Ziv frames are quantized after removing redundancy, and a parity syndrome is generated and transmitted by the SW encoder. The SW decoder first decodes the Key frame and then obtains a frame between the decoded information by motion compensation. The image sequence obtained in this way is called side information. The SW decoder performs error correction on the components that could not be predicted by the side information using the parity syndrome. If the correction fails, the decoder requests additional information from the encoder and receives a longer parity syndrome. The decoded signal is inversely quantized to restore redundancy and output.

(2) Robust super resolution (SR)
An image acquisition model in super-resolution restoration is generally expressed as follows (see Non-Patent Document 2). This model is shown in FIG.

Y _k is a [HW × 1] vector of [H × W] pixel observation images rearranged lexicographically, and X is a [RH 2 RW] pixel high resolution image lexicographically rearranged [R ² HW × 1] vector. k, N, and R are the frame index, the number of observation images, and the resolution ratio of the observation image and the high-resolution image, respectively. D, H _k , and F _k represent matrices that cause down-sampling, blurring, and displacement, respectively. V _k is a vector of additional noise. When it is assumed that the point spread function (PSF) of blurring is uniform throughout the entire image, H _k = H can be substituted. Further, assuming that the displacement is only translational movement, HF _k = F _k H is established. Here, if Z = HX (Z is an image obtained by adding blur to X), the maximum likelihood estimation problem is expressed by the following equation.

Where ‖ ‖ _p is the norm of the loss function. It is difficult to solve equation (2), and it is a general method to obtain an approximate solution using iterative calculation by the gradient method. When solving with the gradient method,

think of. First, when p = 1

It becomes. Here, a D, F _k transposed matrix D ^T, F ^T _k is upsampled respectively, to act in the opposite direction shifts the F _k matrix. Therefore, D, F _k , D ^T , and F ^T _k do not change the value of Z, but only change their positions in the high resolution grid. Equation (4) holds when each pixel Z (i) has the same sign “+1” and “−1” values, ie, the median of the pixel value whose solution is aligned to that point. It means that.

  Next, when p = 2,

It becomes. As before, D, F _k , D ^T , and F ^T _k do not change the value of Z. Equation (5) is established when the sum of the differences of each pixel is equal to 0, and means that the solution of Equation (5) is the average value of the pixel values aligned with that point. The process of using the alignment and the average value or median in this way is called fusion.

  FIG. 3 shows an example of alignment with a high resolution grating. The left side of the figure shows an example of an observation image sequence of [2 × 2] pixels, and the right side shows an example of the result of being positioned on a double high-resolution grid.

  After performing the fusion, a high-resolution image X is obtained by simultaneously performing interpolation and blur removal of pixel values to which the corresponding pixel values are not assigned in the alignment. X is obtained by the following equation.

A is a weight matrix having the number of pixels aligned with Z (i) as diagonal components corresponding to the positions. S ^l _x is the image is l pixel shift the x-axis direction, S ^m _y is an image to m pixel shift the y-axis direction. The second term imposes a constraint that the image is flat. α is a value of 0 ≦ α ≦ 1 that reduces the weight as the distance from the target pixel increases, and λ is a regularization parameter. Equation (6) is solved by iterative calculation.

β is a step size parameter.

  Both fusion and blur removal + interpolation calculations can be performed at high speed. Also, the average value and median in the fusion step differ in robustness against the calculation speed and outliers. The average value can be calculated faster, and the median processing requires more time as the number of samples increases according to the algorithm. The median is more robust against outliers. The average value is affected by outliers, but the median is not affected even if half of the samples are outliers.

B. Girod, A.M.Aaron, S. Rane, and D. Rebollo-Monedero, "Distributed video coding", Proceedings of the IEEE, Vol. 93, No. 1, pp. 71-83, 2004. S. Farsiu, M.D.Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super resolution", IEEE Transactions on Image Processing, Vol. 13, No. 10, pp. 1327 -1344, 2004.

  In general distributed video coding (DVC), Key frames are transmitted at intervals of a plurality of frames, and Wyner-Ziv frames that are not transmitted are estimated from the Key frames. Since the decoder cannot know the information of the current signal of the Wyner-Ziv frame, how to increase the estimation accuracy is important for improving the coding efficiency. One method is to recursively decode from low frequency to high frequency, and use the low frequency signal of the Wyner-Ziv frame decoded in the process for estimation of the high frequency band. However, interpolation methods such as bicubic interpolation that are generally used do not involve estimation of high-frequency components, and thus are not effective.

  The present invention has been made in view of this problem, and an object of the present invention is to use super resolution (SR) within the framework of the DVC to increase the encoding efficiency of the DVC. Since the super-resolution method enlarges an image while estimating a high-frequency component from a plurality of images, it is considered that a high-resolution image can be accurately estimated even if the key frame is reduced and sent.

  The SR reconstruction method described above achieves robustness by using the median value of the pixel values aligned with a certain pixel for super-resolution. This method is effective when super-resolution is performed using a sufficiently large number of images, but is not robust when there are not many images. This is because if only one erroneous pixel value is aligned with a certain high-resolution grid, the data will be used for super-resolution. To solve this problem, we must consider whether the aligned pixel values are reliable. In the present invention, it is determined whether or not the alignment obtained in pixel units at the time of image alignment is reliable, and the pixels used for super-resolution and the other pixels are divided according to the result. Furthermore, we propose a reconstruction method that increases the rate of contribution to super-resolution for pixels with higher reliability.

  As described above, an object of the present invention is to provide a decoding apparatus and a decoding method that perform super-resolvable region determination for the purpose of improving the encoding efficiency of distributed encoding.

The decoding device of the present invention
A key frame decoding unit that receives and decodes a key frame of an image sequence of a low frequency component from the encoding device;
A flat part determination unit that determines a flat part and a non-flat part of a decoded image sequence, and requests data of a part determined to be a non-flat part from the encoding device;
A non-flat part decoding unit that receives and decodes the data of the non-flat part;
Using the decoded non-flat portion data, the degree of similarity between images is calculated, and the image decoded by applying the super-resolution method to the non-flat portion pixels whose similarity is equal to or greater than the threshold value A super-resolution processing unit that estimates high-frequency components of
It is characterized by having.

The decoding method of the present invention includes:
A decoding method in a decoding device for receiving and decoding an image sequence from an encoding device,
A key frame decoding step for receiving and decoding a key frame of a low-frequency component image sequence from the encoding device;
A flat part determining step of determining a flat part and a non-flat part of the decoded image sequence, and requesting data of a part determined to be a non-flat part to the encoding device;
A non-flat portion decoding step for receiving and decoding the non-flat portion data;
Using the decoded non-flat portion data, the degree of similarity between images is calculated, and the image decoded by applying the super-resolution method to the non-flat portion pixels whose similarity is equal to or greater than the threshold value A super-resolution processing step for estimating a high-frequency component of
It is characterized by having.

  According to the present invention, the coding efficiency of distributed coding can be improved.

Block diagram of DVC encoder / decoder Diagram showing image acquisition model in super-resolution restoration Diagram showing an example of alignment with a high resolution grid Configuration diagram of encoding apparatus / decoding apparatus according to an embodiment of the present invention Diagram showing a 4-neighbor Laplacian kernel Illustration for explaining the similarity in a video

  Examples of the present invention will be described in detail below.

  FIG. 4 shows a configuration diagram of the encoding device 10 and the decoding device 15 according to the embodiment of the present invention.

  The embodiment of the present invention is directed to transmission of an image sequence at a low bit rate. Specifically, most of the high-frequency component information is not transmitted in consideration of being reduced in the compression process. For this reason, the encoding apparatus 10 includes a low-frequency component image sequence acquisition unit 101 that acquires an image sequence of low-frequency components by removing an image sequence of high-frequency components.

The low frequency component image sequence acquisition unit 101 applies a video to be transmitted to a low pass filter to acquire a low frequency component. As an example, there is a Gaussian filter having a size of 3 × 3 and a standard deviation σ of 0.8 as a low-pass filter. The low frequency component image sequence is downsampled at rate r. Hereinafter, this downsampled image sequence will be referred to as a low resolution image sequence _Yk .

  In the embodiment of the present invention, in order to compensate the information of the high frequency component, the decoding device 15 estimates the high frequency component with high accuracy by SR, and raises the quality of the decoded image.

  The low resolution image sequence is divided into odd frames and even frames by the frame dividing unit 103, and a key frame (Key frame) that is an odd frame is transmitted from the Key frame encoding unit 105.

In the decoding device 15, the Key frame decoding unit 151 decodes the Key frame, and the motion estimation unit 153 calculates a motion vector between adjacent Key frames. When the motion vector from Y _2k-1 to Y _{2k + 1} is m ^f _k and the motion vector from Y _{2k + 1} to Y _2k-1 is m ^b _k , the following relationship is established.

The Wyner-Ziv frame is interpolated by bidirectional motion compensation using the motion vector and the Key frame obtained in this way. If the low-resolution image sequence obtained by the decoding device 15 is Y ′ _k , the result is as follows.

Through the above process, the decoding device 15 acquires the same number of image sequences as the encoding device 10. Next, the decoding apparatus 15 divides the flat portion determination unit 155 into a non-flat portion and a flat portion where super-resolution is effective from Y ′. This is because the alignment is inaccurate in the flat portion which is a region having a poor texture. Moreover, since such a region has few high frequency components, simple interpolation is sufficient. The flat part determining unit 155 requests the encoding device 10 for data of the portion determined to be a non-flat part (data corresponding to the non-flat part of the even frame) and receives the data from the non-flat part encoding unit 107 of the encoding device 10. . This is because an accurate low resolution image is required for super-resolution. In the decoding device 15, the non-flat portion decoding unit 157 decodes the data of the non-flat portion, and the enlargement processing unit 159 applies the super-resolution method to the non-flat portion to enlarge the image. The enlargement processing unit 159 performs simple interpolation on the flat portion.

<Registration>
Next, registration in the motion estimation unit 153 will be described.

  Registration in the motion estimation unit 153 is performed using block matching in the spatial domain and a phase-only correlation method. The phase-only correlation method has a problem that the position shift can be estimated with high accuracy but the calculation amount is large and the search area is small. That is, the phase-only correlation method is not suitable when an object moves greatly between frames. In order to solve this problem, in the embodiment of the present invention, first, a rough motion is calculated by block matching of luminance values, and an accurate positional deviation amount is calculated by a phase-only correlation method.

Consider two images F (x, y) and G (x, y) out of K images used for super-resolution, and G (x, y) expresses F (x, y) as m _x ( _{x, y), m y (} x, y) is assumed to have moved only in parallel. Ie

Suppose F is a super-resolution frame. _{m x (x, y),} m y (x, y) but is a real number, considered separately this into an integer part and a fraction part.

m ^int can be obtained by simple space domain block matching. Next, the position shift at the sub-pixel is obtained for each pixel. A small area of N ₁ × N ₂ is extracted from F and G, respectively, and obtained by POC (Picture Order Count). Since displacement at the integer level is required,

And the subpixel positional deviation between f and g may be obtained.

  Since the positional deviation obtained in this way is accurate, the flat portion determination in the flat portion determination unit 155 can be performed with high accuracy.

It is not necessary to calculate the similarity value α _k (x, y) between images if only the positional deviation is _obtained. However, since the similarity value between images is used by the enlargement processing unit 159, it is more specific. A typical calculation method will be described. If G (x, y) is the kth image, the pixel of interest is (x, y), and the POC maximum value of f and g is r (n ₁ , n ₂ ), the similarity value α _k (x, y) is given by the following equation.

<Flat portion judgment>
Next, the flat part determination in the flat part determination part 155 will be described.

  It is difficult to estimate a portion having a poor texture in an image with high accuracy of alignment, and conversely, a region having a lot of texture has a high accuracy. Accurate positional deviation estimation is indispensable for super-resolution. In addition, the flat area can be accurately enlarged even by simple interpolation. Considering these things, it is effective to determine the flat portion and switch the enlargement method according to the result. Various methods are conceivable for determining a flat portion of the image. As a conventional technique, there is a method in which the standard deviation of the luminance value of an image block is calculated, and when it is larger than a threshold value, a non-flat portion is obtained. However, with this method, in an image in which a portion with a high luminance value and a portion with a low luminance value are mixed, a flat portion may be determined as a non-flat portion.

  In the embodiment of the present invention, a region having a low high-frequency component is defined as a flat portion. That is, an edge of an image is extracted, and an area with less energy is a flat portion. The flat part determination unit 155 applies a high-pass filter to the image sequence Y ′ generated from the Key frame and input from the motion estimation unit 153 to obtain an edge image sequence W.

Here, C is a 4-neighbor Laplacian kernel shown in FIG.

Next, the obtained image sequence is processed for each N ₁ × N ₂ block. Now, let the block of interest in the edge image sequence W be w (x, y). Where x = 1, ..., N and y = 1, ..., N. A flat portion and a non-flat portion are determined by performing threshold processing on an average value of absolute values of luminance values of the block of interest. A numerical value representing non-flatness,

Define as N ₁ and N ₂ are the height and width of the region of interest. When this value is larger than the threshold value, it is determined as a non-flat portion, and the flat portion determination unit 155 requests data from the encoding device 10. As described above, the flat part determination unit 155 receives the sub-pixel positional deviation from the motion estimation unit 153, thereby enabling the flat part determination with higher accuracy than the normal positional deviation is received. The non-flat part encoding unit 107 of the encoding device 10 transmits the requested data to the encoding device 15.

<Selective SR in pixel units>
Next, the selective SR for each pixel in the enlargement processing unit 159 will be described.

  If a sequence that greatly changes in time locally is super-resolved, an undesired pixel value is used, and an unnatural image is generated. In the embodiment of the present invention, the similarity at the time of alignment is used to avoid this. As described above, the decoding device 15 acquires the non-flat portion data from the encoding device 10, but further increases the effect of super-resolution by using the similarity in the acquired non-flat portion data. be able to.

  FIG. 6 is a diagram showing a frame in which a woman blinks from the video Suzie. Paying attention to the eyes, (a) and (b) are not similar, but (a) and (c) are similar. When the similarity values of (a) and (b) and the similarity values of (a) and (c) are calculated according to the equation (13), the former is 8.7 and the latter is 9.9. For this reason, it is preferable that data with high similarity is used for super-resolution, and data with low similarity is not used for super-resolution. In the embodiment of the present invention, the higher the degree of similarity, the greater the contribution to super-resolution, and the data whose degree of similarity is lower than the threshold value is not used for super-resolution. This can be achieved by changing A which is the weighting matrix of equation (6). In the conventional method, when the low-resolution image is aligned with the high-resolution grid, the value of the element A (x, y) corresponding to the position of the pixel value of the alignment destination is uniformly increased. Then, by adding the calculated similarity, the contribution to the super-resolution proportional to the similarity is increased.

Here, r is an enlargement factor and round is a rounding function.

<Effect of the embodiment of the present invention>
As described above, according to the embodiment of the present invention, high encoding efficiency of DVC can be realized, and the quality of a restored image can be improved. In addition, according to the embodiment of the present invention, it is possible to accurately estimate the amount of displacement that is indispensable for the super-resolution method, and to realize the robustness of the super-resolution method.

  For convenience of explanation, the encoding device / decoding device according to the embodiment of the present invention has been described using a functional block diagram, but the encoding device / decoding device of the present invention may be hardware, software, or A combination thereof may be realized. For example, each functional unit of the encoding device / decoding device may be realized by software, and may be realized as a program in the encoding device / decoding device. In addition, two or more embodiments and each component of the embodiments may be used in combination as necessary.

  As mentioned above, although the Example of this invention was described, this invention is not limited to said Example, A various change and application are possible within a claim.

DESCRIPTION OF SYMBOLS 10 Encoding apparatus 101 Low frequency component image sequence acquisition part 103 Frame division part 105 Key frame encoding part 107 Non flat part encoding part 15 Encoding apparatus 151 Key frame decoding part 153 Motion estimation part 155 Flat part determination part 157 Non Flat part encoding part 159 Enlargement processing part

Claims (5)

A decoding device that receives and decodes an image sequence from an encoding device,
A key frame decoding unit that receives and decodes a key frame of an image sequence of a low frequency component from the encoding device;
A flat part determination unit that determines a flat part and a non-flat part of a decoded image sequence, and requests data of a part determined to be a non-flat part from the encoding device;
A non-flat part decoding unit that receives and decodes the data of the non-flat part;
Using the decoded non-flat portion data, the degree of similarity between images is calculated, and the image decoded by applying the super-resolution method to the non-flat portion pixels whose similarity is equal to or greater than the threshold value A super-resolution processing unit that estimates high-frequency components of
A decoding device. The flat portion determination unit, by comparing the average value and the threshold value of the absolute value of the luminance value of a predetermined block in the edge image sequence obtained by multiplying the high-pass filter to the image sequence, the magnitude of the high frequency component determining a flat portion and a non-flat portion, the decoding apparatus according to claim 1. Performing motion estimation using block matching in the spatial domain, further further comprising a motion estimator for motion estimation using the phase-only correlation method, decoding apparatus according to claim 1 or 2. A decoding method in a decoding device for receiving and decoding an image sequence from an encoding device,
A key frame decoding step for receiving and decoding a key frame of a low-frequency component image sequence from the encoding device;
A flat part determining step of determining a flat part and a non-flat part of the decoded image sequence, and requesting data of a part determined to be a non-flat part to the encoding device;
A non-flat portion decoding step for receiving and decoding the non-flat portion data;
Using the decoded non-flat portion data, the degree of similarity between images is calculated, and the image decoded by applying the super-resolution method to the non-flat portion pixels whose similarity is equal to or greater than the threshold value A super-resolution processing step for estimating a high-frequency component of
A decoding method comprising: In the flat portion determination step, the magnitude of the high-frequency component is determined by comparing an average value of absolute values of luminance values of a predetermined block in the edge image sequence obtained by applying a high-pass filter to the image sequence and a threshold value The decoding method according to claim 4, wherein the flat part and the non-flat part are determined by the following.