KR100941640B1 - Mac based video coder and method of coding a video in the same - Google Patents

Abstract

The present invention relates to a MAC-based image encoding apparatus for compensating for a variation by reflecting a disparity map reconstructed so that an error does not occur in encoding and decoding, and calculating a motion vector in consideration of the disparity map. The MAC-based image encoding apparatus includes a disparity estimator, a MAC encoder, and a disparity compensator. The disparity estimator estimates the disparity using the first image and the second image and outputs a disparity map. The MAC encoder encodes the disparity map output from the disparity estimator, and restores and outputs the encoded disparity map. The disparity compensator compensates for the disparity based on the outputted reconstructed disparity map, and transmits a residual image having information about the compensated disparity to the MAC encoder. In this case, since the MAC-based image encoding apparatus compensates for the variation by using the disparity map reconstructed by the MAC encoder, an error that may occur in the disparity map during encoding is reflected in the residual image. The image may be reconstructed.

Disparity Map, Image, Estimation, Coding, Residual Image

Description

MAC-based video encoding apparatus and a video encoding method therefor {MAC BASED VIDEO CODER AND METHOD OF CODING A VIDEO IN THE SAME}

The present invention relates to a MAC-based video encoding apparatus and an image encoding method. More particularly, the present invention relates to a MAC-based video encoding apparatus, and more particularly, to compensate for a variation by reflecting a reconstructed variation map so that no error occurs during encoding and decoding, and to calculate a motion vector in consideration of the variation map. The present invention relates to a MAC-based video encoding apparatus and a video encoding method.

The MAC-based image encoding apparatus is an apparatus for encoding a 2D image photographed by a camera.

1 is a block diagram illustrating a conventional MAC-based video encoding apparatus.

Referring to FIG. 1, a conventional MAC-based image encoding apparatus includes a variation estimator 100, an encoder 102, a MAC encoder 104, and a variation compensator 106.

The disparity estimator 100 estimates the disparity using the left image and the right image photographed by the camera, and generates a disparity map having an estimation result.

The encoder 102 encodes a left image and calculates a motion vector using the reference image and the left image.

The MAC encoder 104 encodes the disparity map using the motion vector output from the encoder 102 and encodes the residual image output from the disparity compensator 106.

The disparity compensator 106 compensates for the disparity using the disparity map provided from the disparity estimator 100 and outputs the residual image having the compensation result.

In the conventional MAC-based video encoding apparatus, an error frequently occurs during the encoding of the disparity map. In this case, since the error generated in the encoding process is not reflected in the disparity compensation process, the encoding error is not reflected in the residual image. As a result, there is a problem that the desired right image is not restored when the right image is reconstructed using the disparity map and the residual image.

Ideally, the motion vector of the left image should match the motion vector of the disparity map. In this situation, since the motion vector is calculated considering only the left image, an error may occur in the DCT and quantization process for encoding the disparity map.

In addition, since the conventional MAC-based image encoding apparatus estimates the variation by applying the same window to the entire image, blurring may occur in the boundary region.

It is an object of the present invention to provide a MAC-based video encoding apparatus and an image encoding method in which a variation is accurately estimated and an error is prevented from occurring when encoding and decoding.

In order to achieve the above object, the MAC-based video encoding apparatus according to an embodiment of the present invention includes a disparity estimator, a MAC encoder and a disparity compensator. The disparity estimator estimates the disparity using the first image and the second image and outputs a disparity map. The MAC encoder encodes the disparity map output from the disparity estimator, and restores and outputs the encoded disparity map. The disparity compensator compensates for the disparity based on the outputted reconstructed disparity map, and transmits a residual image having information about the compensated disparity to the MAC encoder.

The MAC-based image encoding apparatus according to another embodiment of the present invention includes a variation estimator, an encoder, and a MAC encoder. The disparity estimator compares the first image and the second image, estimates the disparity, and outputs a disparity map. The encoder receives one of the images as a reference image, receives the disparity map, calculates a motion vector based on the received reference image and the disparity map, and outputs the calculated motion vector. The MAC encoder encodes and outputs the output disparity map based on the output motion vector.

An image encoding method according to an embodiment of the present invention comprises the steps of outputting a disparity map by estimating the disparity using a first image and a second image; Encoding the disparity map and restoring the encoded disparity map; And compensating for the variation based on the reconstructed disparity map to output a residual image.

Since the MAC-based image encoding apparatus and the image encoding method according to the present invention compensate for the variation by using the disparity map restored by the MAC encoder, an error that may occur in the disparity map during encoding is reflected in the residual image. Therefore, there is an advantage that the desired image can be restored when the image is restored after decoding.

In addition, since the MAC-based image encoding apparatus and the image encoding method according to the present invention calculate a motion vector in consideration of the disparity map, an error may not occur when the MAC encoder encodes the disparity map using the motion vector. There is an advantage.

In addition, since the MAC-based image encoding apparatus and the image encoding method according to the present invention smooth the variation in the estimation of the variation, an error may not occur in the DCT and the quantization process when encoding the variation map.

Furthermore, since the MAC-based image encoding apparatus and the image encoding method according to the present invention estimate the variation after dividing a block in the boundary region into sub-blocks, the variation value of the boundary region can be accurately expressed, and thus the variation estimation is performed. There is an advantage that the contour of the subject can be accurately represented.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram illustrating a MAC-based video encoding apparatus according to a first embodiment according to an embodiment of the present invention.

Referring to FIG. 2, the MAC-based image encoding apparatus of the present embodiment includes a variation estimator 200, an encoder 202, a MAC encoder 204, and a variation compensator 206.

The variation estimator 200 receives, for example, a left image and a right image generated by using a stereo camera, estimates the variation using the received left image and the right image, and generates the resulting variation. Print a Disparity map. In addition, the disparity estimator 200 smoothes the disparity map as described later and divides the boundary block to estimate disparity. Here, one of the left image and the right image is selected as the reference image. In the following description, it is assumed that the left image is a reference image for convenience of explanation. In addition, the variation means that the distance in which the same pixels are separated between the two viewpoint images is expressed in pixel units.

An encoder 202 receives the left image and the disparity map, and generates a motion vector based on the left image and the disparity map. In addition, the encoder 202 encodes the left image and outputs an encoded signal S1 having information about the encoded left image.

The MAC encoder 204 is a MAC-based encoder that encodes the disparity map output from the disparity estimator 200 using the motion vector provided from the encoder 202, and restores and outputs the encoded disparity map. Let's do it.

The disparity compensator 206 compensates for the disparity based on the reconstructed disparity map output from the MAC encoder 204, and provides a residual image having the compensation result to the MAC encoder 204.

The MAC encoder 204 encodes the disparity map output from the disparity estimator 200 and the residual image provided from the disparity compensator 206, and includes an encoded signal having information about the encoded disparity map and the residual image. Outputs S2).

Although not shown in FIG. 2, the coded signal S2 is decoded by a decoder (not shown), and uses the left image included in the coded signal S1, the disparity map generated by the decoding, and the residual image. The right image is restored by. Then, the left image and the right image are combined to implement a 3D image.

Although not described in detail above, the present invention has four different features compared to the conventional encoding apparatus.

Below, we will look at these four features in turn.

First, the MAC-based video encoding apparatus of the present embodiment compensates for the variation by using the disparity map restored by the MAC encoder 204. The reason for this compensation method will be described in detail by comparing the present invention with the conventional method.

In the conventional encoding apparatus, the disparity map is provided directly to the disparity compensator without passing through the MAC encoder. Then, the residual image and the disparity map outputted by the disparity compensator were encoded after being encoded by the MAC encoder. However, in the process of encoding the disparity map, an error is generated in the disparity map, so that a predetermined variation included in the disparity map may have an incorrect value. As a result, an error generated in the encoding process cannot be reflected in the residual image generated by using the disparity map before encoding. Thus, after the encoded disparity map and the residual image are decoded, the right image may not be correctly reconstructed when the right image is reconstructed by using the decoded disparity map and the residual image. That is, the desired right image was not restored.

In the MAC-based image encoding apparatus of the present embodiment, the disparity map encoded by the MAC encoder 204 and then reconstructed is provided to the disparity compensator 206. In other words, an error that may be generated by encoding is reflected in the disparity map provided to the disparity compensator 206. Therefore, the error due to the encoding is also reflected in the residual image generated by compensating for the variation based on the disparity map reflecting the error. As a result, when the right image is reconstructed by using the decoded disparity map and the residual image, since the error generated by the encoding is reflected in the disparity map and the residual image, the right image may be correctly reconstructed. .

That is, since an error may occur when encoding the disparity map, the MAC-based image encoding apparatus of the present embodiment generates the residual image by compensating for the disparity based on the disparity map reconstructed so that the desired right image is reconstructed.

Second, the MAC-based image encoding apparatus of the present embodiment generates a motion vector by using a left image and a disparity map together. This motion vector generation method will now be described with reference to the accompanying drawings.

3 is a flowchart illustrating a motion vector generation process according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating a process of obtaining a pixel sum for a motion vector according to an embodiment of the present invention.

Referring to FIG. 3, the encoder 202 obtains a first left image of a reference view, a second left image of a current view, a first disparity map of a reference view, and a second disparity map of the current view (S300). Here, the reference time point may be a time point immediately before the current time point or may be another point in time in the past.

Subsequently, the encoder 202 obtains a first pixel sum of the left image and a second pixel sum of the disparity map (S302). In detail, when it is desired to find the same image as the image of the specific block of the first left image in the second left image, the encoder 202 may determine a specific block among the second left images as shown in FIG. The window corresponding to the specific block is searched while the window is moved from the window W1 corresponding to the corresponding position. For example, assume that the window W1 has been moved to the position of the window W2. In this case, the encoder 202 may include pixels included in the window W1 of the first left image (P11 to P91 in FIG. 4B) and pixels included in the window W2 of the second left image. The pixel sums of the pixels (P12 to P92 in FIG. 4B) are added to obtain the first pixel sum. That is, the first pixel sum is (P11-P12) + (P21-P22) + (P31-P32) + (P41-P42) + (P51-P52) + (P61-P62) + (P71-P72) + (P81-P82) + (P91-P92). Here, P means a pixel value of the pixel. Subsequently, the encoder 202 calculates a second pixel sum by performing the same process on the disparity map at a position synchronized with the position at which the first pixel sum is obtained. Since this process is similar to the process of obtaining the first pixel sum, a description thereof will be omitted.

Subsequently, the encoder 202 calculates a third pixel sum by adding the first pixel sum and the second pixel sum (S304). Here, since the third pixel sum is continuously obtained as the window W1 moves, a plurality of third pixel sums are generated in the searching process.

Then, the encoder 202 compares the sizes of the third pixel sums (S306).

Subsequently, the encoder 202 detects the third pixel sum having the smallest value among the third pixel sums, and selects a window movement shift corresponding to the detected third pixel sum as a motion vector (S308).

In other words, unlike the conventional encoding apparatus in which the window movement shift corresponding to the first pixel sum having the smallest value among the first pixel sums is selected as the motion vector, the MAC-based image encoding apparatus of the present embodiment corresponds to the left image. A window movement variation corresponding to the smallest value among the sum of the first pixel sum and the second pixel sum corresponding to the disparity map is selected as the motion vector. The cost function considering the disparity map in estimating the motion vector is shown in Equation 1 below.

f (u, v, m) = MAE (Icur, Iref, u, v, m) + λ MAE (Dcur, Dref, u, v, m). Here, f (u, v, m) represents a cost function of the variation m at (u, v), and Icur and Iref mean pixel values of the left image of the current view and the left image of the reference view. In addition, Dcur and Dref represent the magnitude of variation at the current time point and the variation at the reference time point, and λ denotes a Lagrange multiplier, and MAE () indicates a mean absolute error.

As can be seen from Equation 1, the MAE-based image encoding apparatus of the present embodiment obtains the cost function by considering the left image and the disparity size, and selects the displaced motion as the motion vector when the cost function of the minimum value is obtained.

In the ideal encoding apparatus, the first pixel sum and the second pixel sum should be equal to each other, but in practice, the first pixel sum and the second pixel sum are different from each other. Nevertheless, the conventional encoding apparatus obtains a motion vector using only the first pixel sum and encodes a disparity map using the obtained motion vector. That is, the motion vector of the conventional encoding apparatus does not reflect the motion of the disparity map. Therefore, an error may occur when the disparity map is encoded using the motion vector.

On the other hand, the MAC-based image encoding apparatus of the present embodiment reflects the movement of the disparity map as well as the movement of the left image when obtaining a motion vector. Therefore, an error may not occur when the MAC-based image encoding apparatus encodes the disparity map using the motion vector.

Third, we will look at the process of smoothing variation in estimating variation. This smoothing process is a process that did not exist in the conventional encoding apparatus.

5 is a flowchart illustrating a variation smoothing process according to an embodiment of the present invention, and FIG. 6 is a diagram illustrating a block movement and block estimation process for variation smoothing according to an embodiment of the present invention. However, in FIG. 6, the part except for the person 600 is assumed to be a curtain.

Referring to FIG. 5, the variation estimator 200 selects a window W1 including a specific block 602 and blocks adjacent to the specific block 602 as shown in FIG. 6 after completing the variation estimation. (S500). Here, since the disparity estimation is performed in units of blocks, the pixels included in each block have the same disparity value. According to another embodiment of the present invention, the window W1 may be selected as another range, that is, there is no particular limitation on the window selection method.

Next, the variation estimator 200 selects a variation value Mk (S502). In detail, the variation estimator 200 arranges the variation values as shown in Table 1 below, and selects one of the variation values.

M0 M1 M2 M3 M4 M5 M6 M7 M8 5 3 2 2 2 2 2 2 3

According to an embodiment of the present invention, the variation estimator 200 selects a value located in the middle of the variation values arranged in Table 1 as the initial variation value using median filtering. That is, the variation estimator 200 selects M4 (2) as the initial variation value.

Subsequently, the variation estimator 200 obtains an initial error and a median error (S504). In detail, the disparity estimator 200 obtains pixel values of the specific block 602 in consideration of the original disparity value 5 in the right image of FIG. 6A, and calculates pixel values of the specific block 602 in the left image. Then, the initial error is obtained by summing differences of pixel values for each pixel. In addition, the disparity estimator 200 obtains a pixel value of the specific block 602 in consideration of the selected disparity value 2 in the right image of FIG. 6B, and compares the pixel value with the pixel value of the specific block 602 in the left image. The pixel value difference for each pixel is obtained, and then the median error is obtained by summing the pixel value differences for each pixel. In general, the median error will have a larger value than the initial error since the error value corresponding to the original variation value 5 was the smallest value in the variation estimation.

Subsequently, the variation estimator 200 determines whether a difference between the median error value and the initial error value is within a predetermined range, for example, 5% (S506).

If the difference between the median error value and the initial error value is within the 5% range, the variation estimator 200 changes the variation value 5 of the specific block 602 to the selected variation value 2 (S508). .

On the other hand, if the difference between the median error value and the initial error value is out of the 5% range, the variation estimator 200 determines whether the selected variation value is the last variation value (S510).

If the selected variation value is not the last variation value, another variation value is selected (S502). For example, since there are still many variations after M4 is selected, the variation of M3 or M5 is selected after M4. Then, step S510 is performed again from step S502. That is, the variation estimator 200 continuously changes the variation until the difference between the median error value and the initial error value is within 5%.

However, when the difference between the median error value and the initial error value is 5% or more even after the difference between the median error value and the initial error value is obtained for all the variation values M1 to M8, the variation estimator 200 Maintains the initial shift value 5 as it is (S512).

In short, the variation estimator 200 of the present embodiment changes the variation value of a specific block by using the difference between the median error value and the initial error value, that is, smooths the variation. For example, when the window W is selected for the same curtain as shown in FIG. 6, the variation value of the particular block 602 should not have a large difference from the peripheral variation value. Nevertheless, an error in estimating the variation has caused a particular block 602 to have a significantly larger variation than the surrounding variation. As a result, the high frequency component is included in the disparity map, so that an error may occur in the DCT and quantization process for encoding the disparity map. Thus, there was a need to detect and correct this false block 602, so the present invention smooths the variation of this block 602 using the difference between the median error value and the initial error value.

Fourth, we will look at the process of partitioning blocks in estimating variation. This division process is a process that has not been possible in the conventional encoding apparatus.

7 is a flowchart illustrating a block partitioning process according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating an actual implementation of block partitioning according to an embodiment of the present invention.

Referring to FIG. 7, for example, it is determined whether the specific block 800 is a block of the boundary area (S700).

If the specific block 800 is not a block of the boundary area, the variation of the specific block 800 is maintained as it is (S702).

On the other hand, if the specific block 800 is a block of the boundary region, for example, the specific block 800, which was an 8x8 block, is divided into 4x4 blocks (S704). This is to give more precise variation in the boundary area.

Subsequently, it is determined whether or not the 4x4 block error value is greater than a specific ratio of the 8x8 block error value, for example, 90% (S708). In detail, the disparity estimator 200 divides the specific block 800 into 4 × 4 blocks as shown in FIG. 8 after the smoothing process is completed. Then, the variation estimator 200 estimates error values by applying the variation values around the 4 × 4 blocks instead of the existing variation value 5, that is, obtains the pixel values of the internal pixels. Next, it is determined whether the error values thus estimated are larger than 90% of the 8x8 block values.

If the 4x4 block error value is not greater than 90% of the 8x8 block error value, the variation estimator 200 divides the specific block 800 into 4x4 blocks, and the 4x4 blocks are different from each other. A variation value is given (S710). Thereafter, the variation estimator 200 divides the specific block 800 into 4x4 blocks and outputs a variation map reflecting the same, unlike other blocks in which the variation value is set in units of 8x8 blocks.

On the other hand, when the 4x4 block error value is greater than 90% of the 8x8 block error value, the variation estimator 200 divides the 4x4 blocks into 2x2 blocks again (S712).

Subsequently, the disparity estimator 200 performs disparity estimation on 2x2 blocks (S714), divides the specific block 800 into 2x2 blocks, and gives different disparity values (S716). .

In other words, unlike the other blocks, the MAC-based video encoding apparatus of the present embodiment divides a block into sub-blocks and assigns a disparity value to the divided sub-blocks, unlike other blocks. As a result, the variation value of the blocks of the boundary region can be represented precisely, and thus the outline of the object can be accurately represented during the variation estimation.

In the above, block division is performed from 8x8 blocks, but block division may be performed from blocks of different sizes.

The embodiments of the present invention described above are disclosed for purposes of illustration, and those skilled in the art having ordinary knowledge of the present invention may make various modifications, changes, and additions within the spirit and scope of the present invention. Should be considered to be within the scope of the following claims.

1 is a block diagram illustrating a conventional MAC-based video encoding apparatus.

2 is a block diagram illustrating an apparatus for encoding a MAC based image according to an embodiment of the present invention.

3 is a flowchart illustrating a motion vector generation process according to an embodiment of the present invention.

4 is a diagram illustrating a process of obtaining a pixel sum for a motion vector according to an embodiment of the present invention.

5 is a flowchart illustrating a smoothing process of variation according to an embodiment of the present invention.

6 is a diagram illustrating a block movement and block estimation process for smoothing the variation according to an embodiment of the present invention.

7 is a flowchart illustrating a block division process according to an embodiment of the present invention.

8 is a diagram illustrating an actual implementation of block partitioning according to an embodiment of the present invention.

Claims (20)

A disparity estimator configured to estimate a disparity using the first image and the second image to output a disparity map; A MAC encoder for encoding the disparity map output from the disparity estimator and restoring and outputting the encoded disparity map; And And a variation compensation unit configured to compensate for the variation based on the output of the restored disparity map and to transmit a residual image having information about the compensated variation to the MAC encoder. The method of claim 1, wherein the variation estimator compares the first image and the second image to estimate a first variation and performs a smoothing process of changing a variation of a predetermined block among the first estimated variation. MAC based video encoding apparatus. The apparatus of claim 2, wherein the disparity estimator smoothes the disparity using median filtering. The apparatus of claim 2, wherein the disparity estimator divides the boundary block corresponding to the boundary region among the blocks corresponding to the smoothing process, into subblocks. The apparatus of claim 4, wherein the disparity estimator divides the boundary block into 2 × 2 blocks. The apparatus of claim 1, wherein the MAC-based video encoding apparatus comprises: And selecting an image, receiving the disparity map, obtaining a motion vector based on the selected image and the disparity map, and transmitting the obtained motion vector to the MAC encoder. MAC based video encoding apparatus. The apparatus of claim 6, wherein the encoder calculates a third pixel sum by adding a first pixel sum of the selected image and a second pixel sum of the disparity map on a specific block basis, and calculates the calculated third pixel sums. The smallest third pixel sum is detected, and the window-based shift corresponding to the detected third pixel sum is selected as a motion vector. A variation estimator configured to compare the first image and the second image, estimate a variation, and output a variation map; An encoder that receives one of the images as a reference image, receives the disparity map, calculates a motion vector based on the received reference image and the disparity map, and outputs the calculated motion vector; And And a MAC encoder for encoding and outputting the output disparity map based on the output motion vector. The method of claim 8, wherein the encoder calculates a third pixel sum by adding a first pixel sum of the selected image and a second pixel sum of the disparity map in units of specific blocks, and calculates the calculated third pixel sums. The smallest third pixel sum is detected, and the window-based shift corresponding to the detected third pixel sum is selected as a motion vector. Outputting a disparity map by estimating the disparity using the first image and the second image; Encoding the disparity map and restoring the encoded disparity map; And And compensating for the shift based on the reconstructed disparity map to output a residual image. The method of claim 10, wherein outputting the disparity map, Estimating the variation using the first image and the second image; And And smoothing the estimated variation by changing the variation of some of the blocks corresponding to the estimated variation. The method of claim 11, wherein the smoothing step comprises: Estimating the variation of the specific block using the variation of one of the neighboring blocks of the specific block; And Changing the variation of the specific block to the variation of the surroundings used in the estimation when the difference between the error value generated by the variation estimation and the error value resulting from the variation estimation of the specific block is within a predetermined range. A video encoding method. The method of claim 12, wherein estimating the variation of the block comprises: Selecting a variation corresponding to a median value among the variations of the neighboring blocks as an initial variation; And Estimating the variation of the specific block using the selected variation. The method of claim 12, wherein when the difference between the error value caused by the variation estimation and the error value resulting from the variation estimation of the specific block is out of the predetermined range, the variation of the specific block is sequentially used by using another peripheral variation. Estimating to; And If the difference between the error value resulting from the variation estimation and the error value resulting from the variation estimation of the particular block during all the variation estimation is outside the predetermined range, the specific block includes maintaining the original variation value. An image encoding method. The method of claim 12, wherein the predetermined range is 5%. The method of claim 11, wherein outputting the disparity map, Dividing a boundary block corresponding to a boundary area among the blocks having the smoothed variation into sub blocks; Estimating each of the sub-blocks using the variation of a neighboring block; And Determining whether an error value generated by the variation estimation of the subblocks is greater than a predetermined ratio of the error value generated by the variation estimation of the boundary block; And Changing the variation of the boundary block to the variation of the sub blocks when the error value generated by the variation estimation of the sub blocks is not greater than a predetermined ratio of the error value generated by the variation estimation of the boundary block. An image encoding method. 17. The method of claim 16, wherein when the error value generated by the variation estimation of the subblocks is larger than a predetermined ratio of the error value caused by the variation estimation of the boundary block, the subblock is divided into smaller blocks. Image coding method. 17. The method of claim 16, wherein the predetermined ratio is 90%. The method of claim 10, wherein the video encoding method comprises: Selecting one of the images and calculating a motion vector based on the selected image and the disparity map. The method of claim 19, wherein the calculating of the motion vector comprises: Estimating motion with respect to the selected image to obtain a pixel sum of the selected image; Estimating motion with respect to the disparity map to obtain a pixel sum for the disparity map; And And selecting a vector when the sum of the pixel sums is the smallest as the motion vector.

Images