JP2007316161A - Super resolution processing method and apparatus using residual interpolation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and apparatus of super resolution processing in which the resolution and quality of an image sequence are improved, and a grid artifact is reduced. <P>SOLUTION: Interpolation is performed to create a high resolution estimated image, alignment is performed to create an aligned image, the high resolution estimated image is smeared and the smeared image and the aligned image are subtracted to create the residual image, the deficient residue is interpolated by using the residual value of a nondeficient peripheral image, a smoothing image and reverse projection image are combined to create an enhancement coefficient, the enhancement coefficient is updated to the high resolution estimated image to create a fresh enhancement coefficient, and thereby the residual value of the deficient image is interpolated from the residual value of the peripheral image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention can be used for all image / video processing, particularly image / video super-resolution processing.

  Super-resolution processing is a technique for generating a high-resolution image from a plurality of low-resolution images. This technique can be used for display devices such as a plasma display panel (PDP), a liquid crystal display (LCD), and a cathode ray tube (CRT) to improve the resolution and quality of an input image sequence to be displayed. When the super-resolution processing is used in a recording device such as a hard disk recorder or a DVD (Digital Versatile Disc) recorder, the resolution and quality of an image to be transcoded or stored can be improved. Super-resolution can also be used for printing purposes. In this case, an image sequence from a camera recorder or digital camera can be processed to generate a single high-resolution / high-quality image. Similarly, when applied to monitoring, the resolution and image quality of the face and vehicle license plates can be improved.

  Super resolution processing methods include methods that involve image alignment. In this method, the observed low-resolution image is motion-compensated and converted into a common high-resolution grid called an alignment image. The first estimated frame of the high resolution frame is obtained using a spatial interpolation method such as bicubic interpolation. This estimated frame is improved by repeating the process of blurring, frame difference, inverse point spread function, regularization, and refinement. The residual image is generated during the operation for obtaining the frame difference. Thereby, the alignment image is subtracted from the blurred high resolution estimation frame. For details of the super-resolution processing, refer to, for example, US Patent Application No. 2004/0156561 A1 and US Patent Application No. 2005/0019000 A1.

  In order to obtain a high-quality super-resolution image, it is important that the alignment image is of good quality. In order to perform accurate alignment, it is necessary to perform accurate motion compensation and sub-pixel shift between the target image and the surrounding image. If no sub-pixel shift is performed, a defective pixel is generated in the alignment image. As a result, similar missing pixels are generated in the residual image. Corresponding missing pixels in the high-resolution estimated image are less refined than those without missing pixels. The problem is that these defective pixels occur periodically, resulting in grid artifacts. For this reason, the edge looks jagged or sawtooth. For example, in the case of double-resolution upsampling, the pattern is an alternating pattern in which the luminance is high, low, or high for each pixel.

  In the present invention, residual interpolation is used to solve the above-mentioned grid artifact problem. In residual interpolation, residual values of missing pixels are interpolated from residual values of neighboring pixels that are not missing.

  The present invention is new in that it uses residual interpolation to prevent grid artifacts. By adding a smoothing constraint condition to the high-resolution estimated image, grid artifacts can be prevented even by regularization. However, since the smoothing constraint condition is applied to the entire image, it is necessary to control the strength of the smoothing. In areas that do not require smoothing, smearing occurs, and if the intensity is not set appropriately, smearing becomes excessive or insufficient.

  Since the residual interpolation is performed on the defective pixel, the region where the pixel is not lost is not blurred. The intensity of residual interpolation is inevitably set according to the value of the residual itself. The larger the residual value, the larger the interpolated value, and vice versa.

  As a result, a superior super-resolution algorithm that can produce a clearer image is obtained compared to the case of using regularization.

  Residual interpolation can be combined with regularization. By using this combination, it is possible to prevent the blurring from being excessively performed by suppressing the regularization strength.

Post-Processing or Pre-Processing Device FIG. 2 is a block diagram illustrating the use of the super-resolution processing device in the post-processing or pre-processing device. The input D252 includes a video and / or audio source. If necessary, the video source is separated from other sources using the demultiplexer 202. If necessary, another source or another bit stream D256 is processed by another processing device. Also, the video source may be compressed. In this case, the encoded video bitstream D258 is decoded by the video decoder 204 to generate a decoded image sequence D262. The video source may not be compressed. In this case, an uncompressed image sequence D254 is used. Optionally, the pre-processing device 206 may be used to process the image sequences D262 and D254 to generate a processed image sequence D264. The super-resolution processing device 208 processes the image sequence to generate a super-resolution image sequence D266. The super-resolution processor 208 also performs residual interpolation. When the video source is compressed, the super-resolution processor can optionally use auxiliary data D260 from the encoded video bitstream. Examples of auxiliary data include motion vectors, quantization parameters, deblocking filter strength, and the like. The super-resolution image sequence may be further processed by the post-processing device 210. The processed super-resolution image sequence D 268 can be displayed using the display device 214. Optionally, it may be stored using the storage device 212. In the storage device 212, compression may be used for storage.

Encoding Device and Decoding Device FIGS. 3a and 3b are two block diagrams showing the use of the super-resolution processing device in the encoding device (encoder) and decoding device (decoder), respectively. In the encoder, the image sequence is intra-coded or inter-coded. The intra prediction processing device 306 performs prediction using information on the target frame. The motion detection processing device 302 performs motion compensation prediction with reference to other frames. As a result, the images obtained from the processing units 302 and 306 are then processed by the transform processing unit 304, the quantization processing unit 308, and then the entropy processing unit 310 to generate an encoded video bitstream. The target frame is reconstructed using a local decoder including an inverse quantization processing device 312, an inverse transformation processing device 314, a motion compensation processing device 318, and an inverse intra prediction processing device 316, and a reference frame used by the motion detection processing device 302 Stored as In the present invention, the super-resolution processing device 320 is used to improve the quality and resolution of the reference frame used by the motion detection processing device 302 and the motion compensation device 318. The super-resolution processor 320 also performs residual interpolation. Optionally, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in the super-resolution processor 320.

  In the decoder, an entropy decoding processing device 340, an inverse quantization processing device 342, an inverse transformation processing device 344, an inverse intra prediction processing device 346, and a motion detection processing device 350 are used to reconstruct an image sequence from the encoded video bitstream. To do. Similar to the encoder, the super-resolution processor 348 is used to improve the quality and resolution of the reference frame. The super-resolution processor 348 also performs residual interpolation. Optionally, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in the super-resolution processor 348. The super-resolution processing device 348 further improves the displayed image.

  The encoder and decoder described are simplified. Currently, there are encoders and decoders that use advanced tools such as deblocking filters to further improve image quality. In the present invention, encoding and decoding need not be performed as shown in FIGS. 3a and 3b. Here is an example of how super-resolution processing can be used in encoders and decoders to improve coding efficiency or picture quality by improving reference frames used for motion detection, motion compensated prediction, and display. It ’s just that.

Inter-layer prediction encoding apparatus and decoding apparatus FIGS. 4a and 4b are two blocks respectively illustrating the use of a super-resolution processing apparatus for intra-layer intra prediction in an encoding apparatus (encoder) and decoding apparatus (decoder). FIG. In order to improve coding efficiency, inter-layer prediction is used for hierarchical coding. As an example of hierarchical coding using inter-layer prediction, there is a working draft of applying scalable extension to ISO / IEC 1496-10, which is also known as scalable video coding. The base layer image sequence is encoded using the base layer encoding processing device 402. The enhancement layer image sequence is encoded using the enhancement layer encoding processing device 412. The enhancement layer coding processing device 412 improves coding efficiency compared with the case of coding only the enhancement layer by reducing redundancy between the enhancement layer and the base layer using inter-layer prediction. . The redundancy between the enhancement layer residual and the base layer residual is reduced using the inter-layer residual prediction processing device 404. The inter-layer motion data prediction processing device 406 is used to reduce redundancy between motion data in the enhancement layer and the base layer. Examples of motion data include partition size and type used for each macroblock, prediction direction, motion vector, and the like. The inter-layer intra prediction processing apparatus 408 uses a base layer intra image or a base layer reconstructed image as an additional reference image for encoding an enhancement layer image. In order to improve these additional reference images, a super-resolution processor 410 is used. The super-resolution processor 410 also performs residual interpolation. Optionally, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in the super-resolution processor 410.

  In the decoder, the base layer bit stream is decoded using the base layer decoding processing device 422. The enhancement layer decoding processing device 432 decodes the enhancement layer bitstream using the inter-layer residual prediction processing device 424, the inter-layer motion data prediction processing device 426, and the inter-layer intra prediction processing device 428. Similar to the encoder, the super-resolution processor 430 is used to improve the reference picture from the inter-layer intra prediction processor 428. The super-resolution processor 430 also performs residual interpolation. As an option, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in the super-resolution processor 430.

Super Resolution Processing Device FIG. 1a is a block diagram showing a processing device (functional block) in the super resolution processing device. The interpolation processing device 110 interpolates a frame that is a target for super-resolution. Examples of the interpolation method include known bilinear interpolation, bicubic interpolation, and Lanczos interpolation. The interpolated image is stored in the high resolution buffer 112. This is the first high resolution estimated image. Image processing is aligned by the processing unit 122. 122 includes a motion detection processing device 102 that calculates a warping vector between a reference frame and a target frame. The alignment processing device 104 performs alignment based on the warping vector. The point spread function processing device 114 blurs the high resolution estimated image in the high resolution buffer 112. The frame difference processing device 106 calculates a difference (residual) between the alignment image and the blurred high-resolution estimated image. The residual interpolation processing device 124 interpolates the residual value of the missing pixel from the residual values of the neighboring pixels that are not missing. The point spread inverse function processor 108 backprojects the difference (residual) onto a high resolution grid. The high resolution estimation processing device 116 generates the enhancement coefficient by combining the backprojected difference and the smoothing constraint condition. Next, the high-resolution estimated image is updated using this enhancement coefficient. Thereby, the repetition of the improvement process is completed for one cycle. This process is repeated using the processing devices 114, 106, 124, 108, 116 and 118. This process may be repeated until a predetermined number of cycles is reached, or may be repeated until the value of the enhancement coefficient becomes a certain threshold value or less. Further, the adaptive improvement process may be performed on a pixel basis, and only specific pixels may be selected and further improvement process may be performed.

  FIG. 1B is a block diagram showing a processing device (functional block) in the super-resolution processing device similar to FIG. 1A. FIG. 1b differs in that illuminance masking and / or illuminance compensation processor 120 and regularization processor 118 are added. These two processing units are optional. The processing device 120 calculates an illuminance compensation value and an illuminance mask used by the alignment processing device 104. The processing device 118 is used to calculate the smoothing constraint condition used by the processing device 116.

Residual interpolation (general case)
FIG. 5 is a flowchart showing residual interpolation for a general case. All pixels in the residual image are scanned. The scanning order is a raster scanning order or a reverse raster scanning order. In raster scanning, the first row is scanned from left to right, then the second row is scanned from left to right, and so on to the last row. In reverse raster scanning, the first column is scanned from top to bottom, then the second column is scanned from top to bottom, and so on to the last column. Other scanning orders may be used. In step S502, it is confirmed whether the target pixel has a residual. That is, it is confirmed whether the target pixel is a defective pixel. If there is no residual, it is checked in step S504 whether there is a residual sufficient for interpolation in the surrounding pixels. For example, if a 2-tap filter is used, there should be two residual values at that location. Alternatively, the interpolation filter may be changed to match the local characteristics of the residual image. For example, normally, a 6-tap AVC filter can be used when there are 6 values, and a 2-tap bilinear filter can be used when there are only 2 values at the edge. If there is a sufficient residual, in step S506, the residual value of the target pixel is interpolated from the surrounding pixels. Step S510 is executed when it is confirmed that the target pixel has a residual in Step S502, or when there is not a peripheral residual sufficient for interpolation in Step S504, or after Step S506. In step S510, it is confirmed whether there are other pixels. If it does not exist, the process ends. If present, in step S508, the next pixel is set as the target pixel in the scanning order, and the process in step S502 is repeated.

  The interpolation residual in the current process is used to repeat the process and interpolate other missing pixels in the next process. It is also possible to perform the first residual interpolation process using a raster scanning and horizontal interpolation filter, and the second residual interpolation process using a reverse raster scanning and vertical interpolation filter. This is useful when a two-dimensional separation type interpolation filter is used.

Residual interpolation (specific example)
FIG. 6b is a flow diagram illustrating residual interpolation for the specific case of a double scale ratio and a 2-tap bilinear filter. FIG. 6 a shows an example of the positions of the residual 602 and the interpolation residual 604 in this specific example. Since the scale ratio is twice, the defective pixels are alternately located. Two-pass processing is performed, horizontal interpolation using raster scanning is performed in the first pass, and vertical interpolation using reverse raster scanning is performed in the second pass. The following steps are performed in each pass. That is, in step S606, it is confirmed whether or not the target pixel has a residual. If not, in step S608, it is confirmed whether the previous pixel and the next pixel have a residual in the scanning order. This is because a 2-tap bilinear interpolation filter is used. If possible, in step S610, the residual value of the target pixel is an average value of the residual value of the previous pixel and the residual value of the next pixel. In step S614, it is confirmed whether there are other pixels. If it does not exist, the process ends. If present, in step S612, the next pixel is set as the target pixel in the scanning order, and the process in step S606 is repeated.

System LSI
Each functional block of the super-resolution processing apparatus using residual interpolation is typically realized as an LSI (Large Scale Integrated) circuit that is an integrated circuit. This LSI may be made into one chip or a plurality of chips. The name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Furthermore, if integrated circuit technology that replaces LSI appears as a result of progress in manufacturing technology, naturally, integration may be performed using this technology. Biotechnology can be applied.

It is a block diagram which shows the processing apparatus in a super-resolution processing apparatus. It is a block diagram which shows the processing apparatus in the super-resolution processing apparatus similar to FIG. 1 a except that the regularization processing apparatus and the illuminance masking and / or illuminance compensation processing apparatus are added. It is a block diagram which shows use of the super-resolution processing apparatus aiming at post-processing or pre-processing. It is a block diagram which shows use of the super-resolution processing apparatus in an encoder. It is a block diagram which shows use of the super-resolution processing apparatus in a decoder. It is a block diagram which shows use of the super-resolution processing apparatus in the encoder using inter-layer prediction. FIG. 6 is a block diagram illustrating the use of a super-resolution processor in a decoder that uses inter-layer prediction. It is a flowchart which shows the residual interpolation which made object the general case. It is a figure which shows the position of the residual and interpolation residual for the specific case of a 2 times scale ratio and a 2-tap bilinear interpolation filter. FIG. 6 is a flowchart showing residual interpolation for a specific case of a double scale ratio and a 2-tap bilinear interpolation filter.

Claims (14)

A method for performing super-resolution processing using residual interpolation on an image sequence,
Interpolating to generate a first high resolution estimated image (110);
Performing registration (122) to generate a registration image;
And repeating the following steps.
Blurring the high-resolution estimated image using a point spread function to generate a blurred image (114)
Subtracting the blurred image and the alignment image to generate a residual image (106)
Interpolating missing residuals from the residual image using residual values of neighboring pixels that are not missing (124)
Backprojecting the residual image using a point spread inverse function to generate a backprojected image (108);
Combining the smoothed image and the backprojected image to generate an enhancement coefficient (116)
Updating the enhancement coefficient to a high resolution estimated image to generate a new high resolution estimated image (112) The method of claim 1, wherein registration is performed to generate a registration image.
Generating a warping vector by performing motion compensation (102);
And (104) performing alignment based on the warping vector and generating an alignment image. The method of claim 1, wherein a residual error is interpolated from the residual image using residual values of neighboring pixels that are not missing,
Checking whether the target pixel has a residual value (S502);
If the target pixel does not have a residual value, a step of checking whether or not the neighboring pixel has a residual sufficient to perform interpolation (S504);
Interpolating the residual value of the target pixel from the residual value of the peripheral pixel when the target pixel does not have a residual value and the peripheral pixel does not have sufficient residual to perform interpolation (S506),
Checking whether there are other pixels (S510);
When there is no other pixel, setting the target pixel as the next pixel in the scanning order (S508);
Repeating the process from the step of checking whether the target pixel has a residual value when no other pixel exists;
And a step of ending the process when it is confirmed that no other pixel exists. The method according to claim 3, wherein a residual error is interpolated from the residual image by using residual values of neighboring pixels that are not missing.
The scanning order is any one of raster scanning, reverse raster scanning, and arbitrary scanning order. The method of claim 1, further comprising interpolating a missing residual from the residual image using residual values of neighboring pixels that are not missing,
A method comprising: repeating the method of claim 3 so that a new interpolated residual value can be generated using the interpolated residual value. The method of claim 1, further comprising interpolating a missing residual from the residual image using residual values of neighboring pixels that are not missing,
Using a different scan order for each residual interpolation pass. The method of claim 1, wherein a residual error is interpolated from the residual image using residual values of neighboring pixels that are not missing,
Checking whether the target pixel has a residual value (S606);
A step of checking whether the previous pixel and the next pixel have a residual when the target pixel does not have a residual value (S608);
If the target pixel does not have a residual value, and if the previous pixel and the next pixel have residual values, the residual value of the target pixel is determined as the residual value of the previous pixel and the next pixel. Interpolating from the difference value (S610);
A step (S614) of checking whether or not another pixel exists;
When there is no other pixel, setting the target pixel as the next pixel in the scanning order (S612);
Repeating the process from the step of checking whether the target pixel has a residual value when no other pixel exists;
And a step of ending the process when it is confirmed that no other pixel exists. An apparatus for performing super-resolution processing using residual interpolation on an image sequence,
Means (110) for performing interpolation to generate an initial high-resolution estimated image;
Means (122) for performing alignment and generating an alignment image;
An apparatus comprising: means for repeating the following steps:
Means (114) for blurring the high resolution estimated image using a point spread function and generating a blurred image
Means (106) for generating a residual image by subtracting the blurred image and the alignment image
Means (124) for interpolating the missing residual from the residual image using the residual values of neighboring pixels that are not missing.
Means (108) for backprojecting the residual image using a point spread inverse function to generate a backprojected image
Means for generating an enhancement coefficient by combining the smoothed image and the backprojected image (116)
Means (112) for updating the enhancement coefficient to a high-resolution estimated image and generating a new high-resolution estimated image An LSI (Large Scale Integrated) circuit for performing super-resolution processing using residual interpolation on an image sequence,
A unit (110) for performing interpolation to generate an initial high resolution estimated image;
A unit (122) for performing alignment and generating an alignment image;
A LIS circuit comprising: a unit that repeats the following steps.
A unit (114) for blurring the high-resolution estimated image using a point spread function and generating a blurred image
A unit (106) for generating a residual image by subtracting the blurred image and the alignment image
A unit (124) for interpolating a missing residual from the residual image using residual values of neighboring pixels that are not missing.
A unit (108) for backprojecting the residual image using a point spread inverse function to generate a backprojected image
A unit (116) for generating an enhancement coefficient by combining a smoothed image and the backprojected image
A unit (112) for updating the enhancement coefficient to a high resolution estimated image and generating a new high resolution estimated image A device that performs super-resolution processing using residual interpolation in a pre-processing device or a post-processing device,
Means (202) for demultiplexing the input source as necessary to generate an uncompressed image sequence or encoded video bitstream and other bitstreams;
Means (204) for decoding the encoded video bitstream as necessary, and generating a decoded image sequence and optionally auxiliary data;
Optionally, means (206) for pre-processing the decoded image sequence or the uncompressed image sequence to generate a processed image sequence;
Means (208) for performing super-resolution processing using residual interpolation according to claim 1 to generate a super-resolution image sequence on the processed image sequence;
Optionally, means (210) for post-processing the super-resolution image sequence to generate a processed super-resolution image sequence;
Means (212) for optionally compressing and optionally storing said processed super-resolution image sequence;
Means (214) for optionally displaying the processed super-resolution image sequence. In an encoder, a device that performs super-resolution processing using residual interpolation,
Means (306) for performing intra prediction;
Means (302) for performing motion detection;
Means (304) for performing the conversion;
Means for performing quantization (308);
Means (310) for performing entropy encoding;
Means (312) for performing inverse quantization;
Means (314) for performing inverse transformation;
Means (316) for performing inverse intra prediction;
Means for performing motion compensation (318);
An apparatus comprising: means (320) for performing super-resolution processing using residual interpolation according to claim 1 to a reference image used for motion detection and motion compensation. In the decoder, a device that performs super-resolution processing using residual interpolation,
Means (340) for performing entropy decoding;
Means for performing inverse quantization (342);
Means (344) for performing inverse transformation;
Means for performing inverse intra prediction (346);
Means for performing motion compensation (350);
An apparatus comprising: means (348) for performing super-resolution processing using residual interpolation according to claim 1 to a reference image used for the motion compensation. An apparatus that performs super-resolution processing using residual interpolation in an encoder that uses inter-layer prediction,
Means (402) for encoding the base layer;
Means (412) for encoding the enhancement layer using inter-layer prediction;
Means (404) for performing inter-layer residual prediction between the enhancement layer and the base layer;
Means (406) for performing inter-layer motion data prediction between the enhancement layer and the base layer;
Means (408) for performing intra-layer intra prediction between the enhancement layer and the base layer;
The means for performing super-resolution processing using residual interpolation according to claim 1 to the intra image or decoded image of the base layer used as an additional reference for encoding the enhancement layer (410) An apparatus comprising: An apparatus that performs super-resolution processing using residual interpolation in a decoder that uses inter-layer prediction,
Means (422) for decoding the base layer;
Means (432) for decoding the enhancement layer using inter-layer prediction;
Means (424) for performing inter-layer residual prediction between the enhancement layer and the base layer;
Means (426) for performing inter-layer motion data prediction between the enhancement layer and the base layer;
Means (428) for performing inter-layer intra prediction between the enhancement layer and the base layer;
Means for performing super-resolution processing using residual interpolation according to claim 1 to the intra-layer image or decoded image of the base layer used as an additional reference for decoding the enhancement layer ( 430).

Images