JP2011015262A - Image decoder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that the difference of a luminance level becomes visual distortion at the pixel boundary of a correlation derivation unit when pixels are simply mixed, while there is a method of increasing a resolution feeling by alternately picking up a high resolution image and a low resolution image, deriving correlation between the high resolution image and the low resolution image, segmenting the highly correlated part of the high resolution image, and mixing the pixel with the low resolution image in order to apply scalable video encoding requiring the high resolution image and low resolution image of the same time within the limited performance of an imaging element.SOLUTION: In the image decoder, the luminance level difference between the pixel of a detection motion detection position and a reduced reference image pixel is defined as a correction value, the high resolution image is referred to using position information for which the scaling processing of the motion detection position is performed to the position information of the high resolution image, the luminance level of the reference position pixel is corrected by the correction value, and then a reference image pixel for which the low resolution image is enlarged and a corrected pixel are mixed by a mixing ratio corresponding to a correlation degree and written back to the enlarged reference image.

Description

  The present invention relates to an image processing method for performing resolution improvement processing on a moving image signal using motion detection and motion compensation.

  In a conventional image encoding method typified by the MPEG video method, a screen is divided into predetermined units, and encoding is performed in the divided units. For example, in the MPEG-4 AVC (Advanced Video Coding) method (Non-Patent Document 1), a screen (picture) is processed in units of 16 horizontal pixels and 16 vertical pixels called macroblocks.

  When motion compensation is performed, the macroblock can be divided into rectangular blocks (minimum horizontal 4 pixels and vertical 4 pixels), and motion compensation can be performed using different motion vectors for each block.

  Further, in Non-Patent Document 2, a high-resolution input image is reduced and converted into a low-resolution image, and the low-resolution image is encoded by a first encoding unit and a first encoding unit. A hierarchical moving picture coding method having a second coding unit that performs coding on a difference image between an image obtained by enlarging a locally decoded image with respect to an obtained low resolution image and a high resolution input image It is disclosed.

  Further, Patent Document 1 and Patent Document 2 are cited as embodiments at the prior art level for improving resolution at the time of decoding.

H.264 / AVC ISO / IEC 14496-10 MPEG-4 Advanced Video Coding Standard H.264 / AVC Annex G Scalable Video Coding

JP 2007-96709 A JP 2008-31163 A

  A conventional compounding apparatus will be described using Patent Document 2 as an example.

  FIG. 15 is a block diagram of a conventional image decoding apparatus. In order to decode and reproduce the moving image data encoded by the recording process, first, the entropy decoding unit 1500 decodes the bit string of the encoded data subjected to variable length encoding or arithmetic encoding into a normal code. Among the decoded information, the pixel coefficient information is taken out as pixel information through the inverse DCT means 1501 and the inverse quantization means 1502.

  In the case of intra-frame coding, it is stored in the image frame memory 1503.

  In the case of inter-frame predictive coding, the obtained pixel information is a residual signal at the time of motion compensation, and is already decoded and stored in the image frame memory 1503 according to the motion vector information decoded by the entropy decoding unit 1500. Pixel data read from the reference position of the pixel data is added by the adder 1504 and stored in the image frame memory 1503.

  The moving image data stored in the image frame memory 1503 is sequentially super-resolved (enlarged) and stored in the frame memory 1510.

  On the other hand, the high-resolution still image encoded data imaged and encoded at the same time as some frames of the image data periodically or sporadically is subjected to variable length encoding or entropy decoding by the entropy decoding unit 1505 as described above. The bit string of the arithmetically encoded data is decoded into a normal code, and the pixel coefficient information is stored in the frame memory 1508 as pixel information through the inverse DCT unit 1506 and the inverse quantization unit 1507.

  Subsequently, motion prediction is performed between the image frames in the frame memory 1510 using the image frame at the time corresponding to the above-described high resolution image in the frame memory 1510 as a reference image, and the obtained motion information is used in the frame memory 1508. With reference to the high-resolution image, the image frame in the reference frame memory 1510 is combined with the pixel. In the pixel synthesis, weighting is controlled according to the difference in pixel values to be synthesized, and the higher the resolution, the higher the weight of the high resolution image.

  The synthesized pixels are written back to the frame memory 1510, and after a predetermined number of frames are accumulated, the display readout control means 1514 displays and outputs them in a predetermined display order.

  In the above-described literature method, a high-resolution image is encoded as a difference image from an image obtained by enlarging a locally decoded image of a low-resolution image. When the difference image is encoded, encoding information such as an encoding mode and a motion vector used when the low-resolution image is encoded is used.

  However, the high-resolution image and the low-resolution image need to encode images at the same time, and the first and second encoding units and a plurality of encoding units corresponding to the respective resolutions are necessary, and high processing is required. There has been a problem of causing an increase in performance or hardware.

  In addition, in the case of an apparatus that encodes image information acquired from an image sensor, a general image sensor represented by a CMOS or CCD sensor uses high-resolution image data for still image shooting due to restrictions on image data read performance. Can be taken out at a low frame rate, or lower resolution image data can be taken out continuously for still image shooting than for still images, but high resolution image data cannot be taken out at a high frame rate. It has been difficult to realize a procedure for generating and encoding a low-resolution image at the same time by reducing the image.

  Further, as in the conventional methods of Patent Documents 1 and 2, there is a visual step at the pixel boundary of the motion compensation unit simply by controlling and combining the high-resolution image and the low-resolution image using motion compensation. There was a problem that occurred.

(Derivation of correlation)
The degree of correlation is derived between the reduced reference picture obtained by reducing the high resolution picture and the low resolution picture, and the pixel of the reduced reference picture having the highest correlation with the pixel of the low resolution picture is mixed according to the degree of correlation. The degree of correlation is derived by performing motion detection with pixels in a predetermined area between the reduced reference image and the low resolution image, and the luminance level difference between the pixel at the motion detection position of the reduced reference image and the pixel of the low resolution image. And the degree of correlation is higher as the absolute value of the luminance level difference is smaller.

  The obtained luminance level difference is used as correction value information for bringing the luminance level of the pixel of the reduced reference image closer to the luminance level of the pixel of the low-resolution image.

(Pixel mixture)
Pixel mixing is performed using the pixels of the enlarged reference image obtained by enlarging the high-resolution image and the low-resolution image, and the enlarged position information obtained by scaling the motion detection position when deriving the correlation to the position information of the high-resolution image. The brightness level of the reference pixel of the high-resolution image that is used and referenced is corrected using the correction value information obtained by deriving the correlation, and the enlarged standard image is correlated with the pixel at the position corresponding to the processing pixel position of the low-resolution image. The pixels are mixed at a mixing ratio according to the above and written back to the enlarged reference image.

1 is a configuration diagram of an image decoding apparatus according to Embodiment 1 of the present invention. Timing chart of image data in Embodiment 1 of the present invention Picture unit flow chart in Embodiment 1 of the present invention Correlation derivation flowchart in embodiment 1 of the present invention Pixel mixing flowchart in Embodiment 1 of the present invention Schematic diagram of pixel reduction processing in Embodiment 1 of the present invention Schematic diagram of pixel enlargement processing in Embodiment 1 of the present invention Schematic diagram of motion detection processing in Embodiment 1 of the present invention Schematic diagram of scaling processing of motion detection results in Embodiment 1 of the present invention Schematic diagram without brightness correction in Embodiment 1 of the present invention Schematic diagram with luminance correction in Embodiment 1 of the present invention Correlation derivation flowchart in the second embodiment of the present invention Schematic diagram of motion detection processing in Embodiment 3 of the present invention Schematic diagram of scaling processing of motion detection results in Embodiment 2 of the present invention Configuration of conventional image decoding device

  Embodiments of the present invention will be described below with reference to the drawings.

  In the configuration of the embodiment of the present invention, the description of the parts common to the processing contents of the conventional image decoding apparatus is omitted.

  FIG. 1 shows an extraction section of high-resolution image data during extraction of low-resolution moving image data within the data extraction capability of the image pickup device in the recording process in addition to the low-resolution moving image data described in FIG. 2 is a configuration diagram of an image decoding apparatus that decodes two encoded data obtained by inserting a low-resolution image and performing high-resolution image encoding independently and in parallel.

  The description will be continued from the state where the frame memory 103 stores the low resolution image and the frame memory 108 stores the high resolution image.

  The high resolution image stored in the frame memory 108 is reduced to the same resolution as the low resolution image by the reduction processing unit 113 and stored in the frame memory 114 as a reference image.

  On the other hand, the low resolution image in the frame memory 103 is converted into the same resolution as the high resolution image in the frame memory 108 by the enlargement processing unit 110 and stored in the frame memory 111.

  Next, the motion detection correlation determination unit 115 performs motion detection processing between the reference image in the frame memory 114 and the processing target image in the frame memory 103. The obtained motion information with low resolution accuracy is scaled with high resolution accuracy, and the corresponding frame in the frame memory 108 is referred to.

  The basic processing flow is that the processing target pixel in the frame memory 111 and the reference pixel in the frame memory 108 are synthesized by the weighting mixing unit 116 and written back to the frame memory 111.

  FIG. 2-200 schematically illustrates temporal arrangement of low-resolution moving image data and high-resolution images.

(Embodiment 1)
The first embodiment will be described in further detail with reference to FIGS. 3, 4, and 5. FIG.

  FIG. 3 shows the flow of processing in units of pixel frames. Following the reference image reduction processing 300 and the standard image enlargement processing 301 to be processed, one frame is divided into predetermined blocks 302, and the degree of correlation in units of blocks. This represents a flow in which the derivation process 303 and the pixel mixing process 304 are loop-processed for the number of blocks.

  FIG. 4 is a flow of processing contents for deriving the correlation. After the motion detection process 400, the difference between the pixel values of the standard image and the reference image is also obtained and the mixing ratio of the pixel mixture is calculated.

  FIG. 5 is a flow of the pixel mixing process, and shows the process from the weighted mixing with the mixing ratio obtained in the correlation deriving process to the writing back to the frame memory.

  FIG. 6 is a schematic diagram of the processing contents of the reference image reduction processing 300, which is an example in the case where the resolution of the high resolution and the low resolution is both horizontal and vertical, and the average of four pixels of the high resolution image is reduced to the low resolution image. The mode of reducing as one pixel is schematically shown.

  FIG. 7 is a schematic diagram of the enlargement process 301 of the reference image to be processed. This is an example in the case where the resolution of the high resolution and the low resolution is double in both horizontal and vertical directions. One pixel of the low resolution image is 2 × 2. This is a schematic illustration of copying and enlarging as 4-pixel equivalence.

  FIG. 8 is a schematic diagram in which the motion detection processing 400 performs motion detection in units of a predetermined rectangular area by the block matching method. FIG. 8 shows a state where the motion vector mv (x, y) indicated by the arrow is obtained with the low resolution pixel accuracy by the processing of Block n.

  FIG. 9 schematically shows the contents of the motion vector scaling processing 501 in the pixel mixture processing 304, and shows a state in which the motion vector has low resolution pixel accuracy even after scaling.

  FIG. 11 schematically shows the contents of the weighted mixture 504 in the pixel mixing process 304, and shows a state in which the pixel values are mixed smoothly by correcting the level difference of the pixel values.

  FIG. 10 shows an example of the case where the correction process of the level difference of the pixel value is not performed, and when there is no correction, there is a luminance level step that causes visual degradation.

(Embodiment 2)
In the second embodiment, as shown in the flow of FIG. 12, a visual improvement effect is obtained for each processing content of the above-described correlation derivation processing 303. Following the motion detection 1200 in units of blocks, The degree of correlation is obtained for each pixel in the motion detection unit block.

(Embodiment 3)
In the third embodiment, as schematically illustrated in FIG. 13, the motion vector detection accuracy of the motion detection processing 400 is set to half pixel accuracy of a low resolution image (integer pixel accuracy of a high resolution image). After the sparse search, the motion vector that appears finally is densely searched around the obtained motion vector using the high-resolution image before the reduction as the reference image, and the resultant pixel is increased to the integer pixel accuracy of the high-resolution image, and then the composition after pixel mixture is performed It improves the visual resolution of the image.

  The image decoding apparatus according to the present invention is a digital video camera or a device having a self-recording / playback function for moving images of a digital still camera, such as a motion detection process that has been operated only during recording originally included in these devices. By utilizing the function at the time of reproduction, it is possible to improve the resolution feeling at the time of moving image reproduction at a very low cost.

100, 105 Entropy decoding means 101, 106 Inverse DCT means 102, 107 Inverse quantization means 103 Image frame memory 108 Image frame memory 111 Image frame memory 114 Image frame memory 104, 109 Adder 110 Enlargement processing means 112 Display reading control means 113 Reduction processing means 115 Motion detection correlation determination means 116 Weighted mixing means with correction 1500, 1505 Entropy decoding means 1501, 1506 Inverse DCT means 1502, 1507 Inverse quantization means 1503 Image frame memory 1504 Adder 1508 Image frame memory 1509 Super-resolution processing Means 1510 Image frame memory 1511 Motion prediction means 1512 Residual detection / combination ratio control means 1513 Weighted mixing means 1514 Display readout control means

Claims (7)

An input is an image data group obtained by decoding encoded data in which an image captured by mixing the first resolution and the second resolution is encoded for each resolution,
Using the reduced image data obtained by converting the image data of the first resolution to the second resolution as reference image data, motion detection is performed for each predetermined pixel unit with the image data of the second resolution,
Using the detection result of the motion detection means, deriving correlation information between the reduced image data and the image data of the second resolution,
Using the correlation information, the enlarged reference image data obtained by converting the image data of the second resolution into the first resolution and the image data of the first resolution are used to output the output image data of the first resolution. An image decoding device to be generated. The image decoding apparatus according to claim 1, wherein the first resolution is higher in definition than the second resolution. The image decoding apparatus according to claim 1, wherein the motion detection is used in a block matching method of a predetermined rectangular area pixel. In the motion detection, reduced image data obtained by converting the image data of the first resolution into the second resolution is used as reference image data, and after a sparse search with integer pixel accuracy with the image data of the second resolution, The image decoding apparatus according to claim 1, wherein a dense search is performed with pixel accuracy of the first resolution. The correlation information includes an absolute value of a difference in luminance level between a pixel at the position of the motion detection result in the reduced image data and a target pixel in the image data of the second resolution as the second resolution unit. The image decoding device according to claim 1, wherein the image decoding device is obtained by: A pixel of the first resolution image data at a position obtained by scaling the detection result of the second resolution obtained by the motion detection according to the first resolution, and a corresponding pixel of the enlarged reference image data When obtaining the image output image data by mixing in units of pixels of the first resolution, the smaller the correlation information, the larger the mixing ratio on the image data side of the first resolution. Item 5. The image decoding device according to Item 4. Pixels of the image data of the first resolution at a position obtained by scaling the detection result of the second resolution obtained by the motion detection according to the first resolution;
When the corresponding pixels of the enlarged reference image data are mixed in pixel units of the first resolution to obtain the image output image data,
The luminance level difference between the pixel at the position of the motion detection result in the reduced image data and the target pixel in the image data of the second resolution is a luminance level correction value.
6. The image decoding apparatus according to claim 5, wherein pixel mixing is performed after the luminance level correction value on the image data side of the first resolution is reduced.