JP2007329678A - Super resolution processing method and apparatus using luminance masking and luminance compensation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and an apparatus for performing super resolution processing with illumination and illumination compensation, to improve super resolution algorithm in a situation whereby there is an illumination difference between images in a sequence of images, and to improve super resolution algorithm in a situation whereby the image sequence is taken from separate cameras with different calibration parameters thus resulting in different tones of levels of brightness. <P>SOLUTION: Disclosed is the method for performing the super resolution processing, and the illumination masking IM and illumination compensation IC are used in combination to securely exclude a block which actually has a motion error from a super resolution. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention can be used for all image / video processing, particularly image / video super-resolution processing.

  A super-resolution processing method for constructing a high resolution (HR) image from a plurality of low resolution (LR) images is disclosed in US Patent Application No. 2004/0156561 A1 (FIG. 1) (Prior Art 1). A similar super-resolution processing method used for compressed image sequences is disclosed in US Patent Application No. 2005/0019000 A1 (FIG. 2) (Prior Art 2). The similarity between prior art 1 and prior art 2 is that the HR estimated image is warped and obscured to form a pseudo LR image, and then the image is subtracted from the observed LR image. Can be mentioned. This warping, obscuring, and subtraction is repeated for each observed LR image involved in the enhancement process. Technical paper by Masayuki Tanaka and Masatoshi Okutomi disclosed in “Reconfiguration-type super-resolution algorithm” Vol. 2004 No. 133, 2004-CVIM-146, November 2004 (FIG. 3) (prior art 3) According to the different method, the observed LR image is aligned and an HR grid including the observed LR pixels is generated. There is no need to warp the HR estimated image. The obfuscation and subtraction are performed for each position where at least one aligned LR pixel exists in the HR estimation image. This has the advantage that a faster algorithm can be obtained.

  As a common step in these prior arts, frame warping is performed. In the prior art 1 and the prior art 2, the warping is performed on the HR estimated image, whereas in the prior art 3, the warping is performed on the observed LR frame. As a method for detecting a warping vector, there are a block matching method, an optical flow method, a mesh affine transformation method, and the like. Of these methods, block matching is known and commonly used. In block matching, an optimal match is searched for by minimizing a distortion function such as an average absolute difference or an average square error. Other factors such as motion field continuity are also added to the distortion function. Block matching can also be performed in the frequency domain using phase-only correlation.

  Because movement in the real world is complex, it cannot be modeled completely by block matching. For example, block matching based on a transformation model is not effective for rotational motion. In addition, since the boundary of the object does not coincide with the boundary of the block, two movements actually occur in one block. In addition, occlusion where a new object hidden in one frame appears in another frame is also a problem of block matching. These errors are called motion errors.

  As a conventional method for minimizing a motion error, there is a method of tagging whether a motion vector is valid or invalid based on a criterion such as a distortion threshold or a difference threshold. If the motion vector is invalid, the block is not involved in the super-resolution algorithm. The block size can be changed so that smaller blocks are used for regions with complex motion. In the super-resolution context, some kind of residual data is added back to the HR estimated image and further refined. Another way to minimize motion error is to test each pixel in the residual and remove pixels that are larger than a certain threshold. This is also known as outlier removal.

  The brightness of the image sequence varies from image to image, such as, for example, flashing light in a photograph, turning on / off a lamp, and natural lighting. Furthermore, the image sequence may be captured by different cameras having different calibration parameters, and as a result, the brightness level and gradation may be different. In such a case, there is a problem that even if the object or the content exactly matches, the motion vector of the corresponding block is tagged as invalid. As a result, information that can be used for super-resolution is lost.

  In the method in which the outlier test is performed only on the residual, an artifact is caused by a change in luminance. This is because the residual is an average value of a plurality of frames, so that the effect of the luminance difference spreads over the plurality of frames. Therefore, there is a possibility that an abnormal value test is passed even for a pixel to be deleted.

  The present invention solves the above-described problems relating to luminance changes and luminance differences using luminance compensation (IC). In the IC, the luminance difference value is added to either the target block or the reference block. As a method of calculating the luminance difference value, there is a method of subtracting the average value of the target block from the average value of the reference block. For the IC, a block size different from the block size used for motion detection can be used.

  In the present invention, luminance masking (IM) is used together with the IC. In IM, a motion error test is performed on a block. This is similar to testing whether a motion vector is valid. However, for IM, a block size different from the block size used for motion detection can be used. For example, assume that motion detection is performed on a 16 × 16 block to generate one motion vector. Next, IM is performed on each 8 × 8 block in the 16 × 16 block. Here, for example, only the upper left 8 × 8 block may be tagged as invalid and the other three 8 × 8 blocks may be valid. This is an advantage for image sequences with complex motion.

  The block sizes for IM and IC may be different from each other. For example, the luminance difference value can be calculated using 16 × 16 blocks, but IM is executed for each 8 × 8 block. The block size used for IM and IC does not need to be a square, and may be a mesh or an arbitrary shape, and may be any size.

  It is also possible to use only luminance masking (IM), which is useful for sequences where there is no luminance change between surrounding frames.

  Also, the choice between combining IC and IM or using IC / IM alone can be made automatically at either the sequence or frame level.

  The present invention is new in that brightness compensation is used in order to solve brightness differences and brightness changes between images used for super-resolution. Using luminance masking in conjunction with luminance compensation ensures that blocks that actually have motion errors are excluded from super-resolution.

  The present invention is also new in that different block sizes can be used for luminance compensation, luminance masking, and motion detection.

  As a result, the super-resolution algorithm can be improved.

  As described above, warping for super-resolution can be performed on either the HR estimated image or the observed LR frame. The present invention is applicable to both methods. In order to generalize the description of the invention, the terms target frame and reference frame were used. Motion detection is performed on a pair of a target frame and a reference frame. For this reason, the reference frame is a warping target frame.

  One conventional method for performing sub-pixel unit motion detection is to upsample a reference frame. In this case, the target frame and the reference frame have different sizes. Regarding motion detection, there is a method of reducing samples when performing block matching. With respect to IM and IC, the present invention performs either upsampling for the target frame / block or downsampling for the reference frame / block. Although the exact method of upsampling or downsampling is not specified here, this does not affect the spirit and scope of the present invention. Alternatively, it is possible to achieve sub-pixel accuracy without performing upsampling of the reference frame by performing motion detection using phase-only correlation.

[Post-processing or pre-processing equipment]
FIG. 6 is a block diagram illustrating the use of a super-resolution processor for post-processing or pre-processing equipment. The input D652 includes a video and / or audio source. If necessary, the demultiplexer 602 is used to separate the video source from other sources. If necessary, another source or another bit stream D656 is processed by another processing device. Also, the video source may be compressed. In this case, the encoded video bit stream D658 is decoded by the video decoder 604, and a decoded image sequence D662 is generated. The video source may not be compressed. In this case, an uncompressed image sequence D654 is used. Optionally, the pre-processing device 606 may be used to process the image sequences D662 and D654 to generate a processed image sequence D664. The super-resolution processor 608 processes the image sequence to generate a super-resolution enhanced image sequence D666. The super-resolution processor 608 also performs luminance masking and / or luminance compensation. When the video source is compressed, the super-resolution processor can optionally use auxiliary data D660 from the encoded video bitstream. Examples of auxiliary data include motion vectors, quantization parameters, deblocking filter strength, and the like. The super-resolution-enhanced image sequence may be further processed by the post-processing device 610. The processed super-resolution-enhanced image sequence D668 can be displayed using the display device 614. Optionally, it may be stored using storage device 612. In the storage device 612, compression may be used for storage.

[Encoder and decoder]
7a and 7b are two block diagrams illustrating the use of the super-resolution processor in the encoder and decoder, respectively. In the encoder, the image sequence is intra-coded or inter-coded. The intra prediction processing device 706 performs prediction using information on the target frame. The motion detection processing device 702 performs motion compensation prediction with reference to other frames. As a result, the images obtained from the processing devices 702 and 706 are then processed by the transform processing device 704, the quantization processing device 708, and then the entropy processing device 710 to generate an encoded video bitstream. The target frame is reconstructed using a local decoder including an inverse quantization processing device 712, an inverse transformation processing device 714, a motion compensation processing device 718, and an inverse intra prediction processing device 716, and a reference frame used by the motion detection processing device 702 Is stored as In the present invention, the super-resolution processing device 720 is used to improve the quality and resolution of the reference frame used by the motion detection processing device 702 and the motion compensation device 718. The super-resolution processor 720 also performs brightness masking and / or brightness compensation. As an option, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in the super-resolution processor 720.

  In the decoder, an entropy decoding processing device 740, an inverse quantization processing device 742, an inverse transformation processing device 744, an inverse intra prediction processing device 746, and a motion detection processing device 750 are used to reconstruct an image sequence from the encoded video bitstream. To do. Similar to the encoder, the super-resolution processor 748 is used to improve the quality and resolution of the reference frame. The super-resolution processor 748 also performs brightness masking and / or brightness compensation. Optionally, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in super-resolution processor 748. The super-resolution processing device 748 further improves the displayed image.

  The encoder and decoder described are simplified. Currently, there are encoders and decoders that use advanced tools such as deblocking filters to further improve image quality. In the present invention, encoding and decoding need not be performed as shown in FIGS. 7a and 7b. Here is an example of how super-resolution processing can be used in encoders and decoders to improve coding efficiency or picture quality by improving reference frames used for motion detection, motion compensated prediction, and display. It ’s just that.

[Inter-layer prediction]
12a and 12b are two block diagrams illustrating the use of the super-resolution processor for inter-layer intra prediction in the encoder and decoder, respectively. In order to improve coding efficiency, inter-layer prediction is used for hierarchical coding. As an example of hierarchical coding using inter-layer prediction, there is a working draft of applying scalable extension to ISO / IEC 1496-10, which is also known as scalable video coding. The base layer image sequence is encoded using the base layer encoding processing device 1202. The enhancement layer image sequence is encoded using the enhancement layer encoding processing device 1212. The enhancement layer coding processing apparatus 1212 improves coding efficiency compared with the case of coding only the enhancement layer by reducing redundancy between the enhancement layer and the base layer using inter-layer prediction. . The redundancy between the enhancement layer residual and the base layer residual is reduced using the inter-layer residual prediction processing device 1204. The inter-layer motion data prediction processing device 1206 is used to reduce redundancy between motion data in the enhancement layer and the base layer. Examples of motion data include partition size and type used for each macroblock, prediction direction, motion vector, and the like. The inter-layer intra prediction processing apparatus 1208 uses a base layer intra image or a base layer reconstructed image as an additional reference image for encoding an enhancement layer image. A super-resolution processor 1210 is used to improve these additional reference images. The super-resolution processor 1210 also performs luminance masking and / or luminance compensation. As an option, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in the super-resolution processor 1210.

  In the decoder, the base layer bit stream is decoded using the base layer decoding processing device 1222. The enhancement layer decoding processing device 1232 decodes the enhancement layer bitstream using the inter-layer residual prediction processing device 1224, the inter-layer motion data prediction processing device 1226, and the inter-layer intra prediction processing device 1228. Similar to the encoder, the super-resolution processor 1230 is used to improve the reference picture from the inter-layer intra prediction processor 1228. The super-resolution processor 1228 also performs luminance masking and / or luminance compensation. As an option, auxiliary data such as motion vectors, quantization parameters, deblocking filter strength, etc. can be used in the super-resolution processor 1228.

[Mode 1 of super-resolution processing apparatus]
FIG. 10 is a block diagram showing a processing device in the super-resolution processing device. The interpolation processing apparatus 1010 interpolates a frame that is a target for super-resolution. Examples of the interpolation method include known bilinear interpolation, bicubic interpolation, and Lanczos interpolation. The interpolated image is stored in the high resolution buffer 1012. This is the first high resolution estimated image. The processing unit 1022 aligns the image sequence. 1022 includes a motion detection processing device 1002 that calculates a warping vector between a reference frame and a target frame and generates a predicted image. The warping vector and the predicted image are input to the luminance masking and / or luminance compensation processing apparatus 1020, and the luminance masking and / or luminance compensation is performed. The alignment processing device 1004 performs alignment based on the warping vector, the luminance mask, and the luminance compensation value. The point spread function processing device 1014 blurs the high-resolution estimated image in the high-resolution buffer 1012. The frame difference processing device 1006 calculates a difference between the alignment image and the blurred high-resolution estimated image. The point spread inverse function processor 1008 backprojects the difference onto a high resolution grid. The regularization processing apparatus 1018 is used to calculate a smoothed image. The high-resolution estimation processing device 1016 generates an enhancement coefficient by combining the backprojected difference and the smoothing constraint condition. Next, the high-resolution estimated image is updated using this enhancement coefficient. Thereby, the repetition of the improvement process is completed for one cycle. This process is repeated using the processing apparatuses 1014, 1006, 1008, 1016, and 1018. This process may be repeated until a predetermined number of cycles is reached, or may be repeated until the value of the enhancement coefficient becomes a certain threshold value or less. Further, the adaptive improvement process may be performed on a pixel basis, and only specific pixels may be selected and further improvement process may be performed.

[Super Resolution Processing Device Mode 2]
FIG. 11 is another block diagram showing a processing device in the super-resolution processing device. The interpolation processing device 1110 interpolates a frame that is a target for super-resolution. Examples of the interpolation method include known bilinear interpolation, bicubic interpolation, and Lanczos interpolation. The interpolated image is stored in the high resolution buffer 1112. This is the first high resolution estimated image. The warping of the image sequence is calculated by the processing device at 1122. 1122 includes a motion detection processing device 1102 that calculates a warping vector between the reference frame and the target frame and generates a predicted image. The warping vector and the predicted image are input to the luminance masking and / or luminance compensation processing device 1120, and luminance masking and / or luminance compensation are performed. The warping processing device 1104 performs warping based on the warping vector, the luminance mask, and the luminance compensation value. The point spread function processing device 1114 blurs the warped image. The frame difference processing device 1106 calculates the difference between the warped image and the observed low resolution image (input image sequence). The point spread inverse function processor 1108 backprojects the difference onto a high resolution grid. Regularization processor 1118 is used to calculate a smoothed image. The high-resolution estimation processing device 1116 generates the enhancement coefficient by combining the backprojected difference and the smoothing constraint condition. Next, the high-resolution estimated image is updated using this enhancement coefficient. Thereby, the repetition of the improvement process is completed for one cycle. This process is repeated using the processing devices 1114, 1106, 1108, 1116, and 1118. This process may be repeated until a predetermined number of cycles is reached, or may be repeated until the value of the enhancement coefficient becomes a certain threshold value or less. Further, the adaptive improvement process may be performed on a pixel basis, and only specific pixels may be selected and further improvement process may be performed.

[Embodiment 1 of luminance compensation and luminance masking]
In the present embodiment, luminance masking and luminance compensation will be described with reference to FIG. The input is a block of the target frame and a block of the reference frame, which are called a target block (cur_blk) and a reference block (ref_blk), respectively. The reference block is a block identified by the motion detection process as the corresponding block that most closely matches the target block. In step S402, the distortion calculator calculates distortion between the reference block and the target block. As a method for calculating distortion, there is a method for calculating an average absolute difference as follows.

          distortion = mean (cur_blk-ref_blk)

  As another method for calculating the distortion, there is a method for calculating the mean square difference as follows.

          distortion = mean ((cur_blk-ref_blk) ^ 2)

  In step S404, the comparator compares the distortion with the threshold value. The threshold may be a fixed threshold, an adaptive threshold based on the characteristics of the previous frame, or an adaptive threshold based on the characteristics of the target frame. It may be a combination of a plurality of threshold values. If the distortion is less than the threshold, or if the distortion meets the threshold criteria, the process ends in step S414 and the corresponding reference block can be used for super-resolution. This information can be stored in memory for later use. On the other hand, if the distortion is greater than the threshold value, or if the distortion does not meet the threshold criteria, step S406 is executed. In step S406, the luminance difference calculator calculates a luminance difference between the target block and the reference block. As a method of calculating the luminance difference, there is a method of calculating an average value difference between the target block and the reference block as follows.

            illum_difference = mean (ref_blk)-mean (cur_blk)

  As another method for calculating the luminance difference, there is a method of finding a difference between DC components by converting a target block and a reference block into a frequency domain. The DCT transform is used as follows, for example. Wavelet transform is also suitable.

illum_difference = Inverse_DCT (DC [DCT (cur_blk)]-
DC [DCT (reference_blk)])

  In step S408, the luminance compensator compensates for the luminance difference. As a method of compensating the difference, there is a method of subtracting all the coefficients in the reference block as follows.

          ref_blk = ref_blk-illum_difference

  As another method, there is a method in which subtraction is performed in the frequency domain and inverse transformation is performed. In step S410, the distortion calculator calculates the distortion as described in step S402. The difference here is that the content of the reference block has changed in step S408. In step S412, the comparator compares the distortion with the threshold value as described in step S404. If the distortion is smaller than the threshold value, or if the distortion meets the threshold criteria, the process ends in step S416. If luminance compensation is used, the corresponding reference block is used for super-resolution. Can do. On the other hand, if the distortion is larger than the threshold value, or if the distortion does not satisfy the threshold criteria, the process ends in step S418, and the corresponding reference block cannot be used for super-resolution. This is also called masking. Again, this information can be stored in memory for later use.

[Embodiment 2 of luminance compensation and luminance masking]
In the present embodiment, luminance masking and luminance compensation will be described with reference to FIG. The processing here is the same as in the first embodiment. In step S502, the distortion calculator calculates distortion as described in step S402. Similar to steps S406, S408, and S410, the luminance difference calculator calculates the luminance difference in step S504, the luminance compensator compensates the luminance difference in step S506, and the distortion calculator calculates distortion in step S508. Steps S406, S408, and S410 are executed only when the distortion does not satisfy the criterion in step S404, whereas steps S504, S506, and S508 are necessarily executed. In step S510, the comparator compares the distortion (not luminance compensated) and the luminance compensated distortion with a threshold value. Again, the threshold may be a fixed threshold, an adaptive threshold based on the characteristics of the previous frame, or an adaptive threshold based on the characteristics of the target frame. There may be a combination of a plurality of threshold values. If neither the distortion nor the brightness compensated distortion is less than a threshold value, or if neither the distortion nor the brightness compensated distortion meets the threshold criteria, the process ends at step S514 and the corresponding reference The block cannot be used for super-resolution. This is also called masking. This information can be stored in memory for later use. On the other hand, if one of the distortion and the luminance compensated distortion satisfies the threshold criterion, the comparator compares the distortion with the luminance compensated distortion in step S512. If the distortion is less than the luminance compensated distortion, the process ends in step S516 and the corresponding reference block can be used for super-resolution. On the other hand, if the distortion is larger than the luminance compensated distortion, the process ends in step S518. If luminance compensation is used, the corresponding reference block can be used for super-resolution. Again, this information can be stored in memory for later use.

[Brightness masking]
In the present embodiment, luminance masking will be described with reference to FIG. This processing is the same as that of the first embodiment except that only luminance masking is used. In step S802, the distortion calculator calculates distortion as described in step S402. In step S804, the comparator compares the distortion with a threshold value. Again, the threshold may be a fixed threshold, an adaptive threshold based on the characteristics of the previous frame, or an adaptive threshold based on the characteristics of the target frame. There may be a combination of a plurality of threshold values. If the distortion is less than the threshold, or if the distortion meets the threshold criteria, the process ends in step S806 and the corresponding reference block can be used for super-resolution. This information can be stored in memory for later use. On the other hand, if the distortion is greater than the threshold value, or if the distortion does not meet the threshold criteria, the process ends in step S808, and the corresponding reference block cannot be used for super-resolution. This is also called masking. Again, this information can be stored in memory for later use.

[Brightness compensation]
In the present embodiment, luminance compensation will be described with reference to FIG. In this process, only luminance compensation is used. Similar to step S406 and step S408, the luminance difference calculator calculates the luminance difference in step S902, and the luminance compensator compensates the luminance difference in step S904.

FIG. 3 is a diagram showing a super-resolution algorithm disclosed in US Patent Application No. 2004/0156561 A1, which is a prior art. FIG. 2 is a diagram showing a super-resolution algorithm disclosed in US Patent Application No. 2005/0019000 A1, which is a prior art. Technical paper by Masayuki Tanaka and Masatoshi Okutomi as prior art “High-speed algorithm for reconstruction-based super-resolution processing” Vol. 2004 No. 133, 2004-CVIM-146, super-resolution processing disclosed in November 2004 It is a figure which shows a model. It is a flowchart which shows a brightness | luminance compensation and a brightness | luminance masking process. It is a flowchart which shows another brightness | luminance compensation and a brightness | luminance masking process. It is a block diagram which shows use of the super-resolution processing apparatus aiming at post-processing or pre-processing. It is a block diagram which shows use of the super-resolution processing apparatus in an encoder. It is a block diagram which shows use of the super-resolution processing apparatus in a decoder. It is a flowchart which shows a brightness | luminance masking process. It is a flowchart which shows a brightness | luminance compensation process. It is a block diagram which shows the processing apparatus in the super-resolution processing apparatus which aligns with respect to the observed low-resolution image sequence. It is a block diagram which shows the processing apparatus in the super-resolution processing apparatus which warps a high resolution estimated image in order to simulate a low resolution image sequence. It is a block diagram which shows use of the super-resolution processing apparatus in the encoder using inter-layer prediction. FIG. 6 is a block diagram illustrating the use of a super-resolution processor in a decoder that uses inter-layer prediction.

Claims (21)

A method for performing super-resolution processing using luminance masking and luminance compensation on an image sequence,
Interpolating to generate a first high resolution estimated image (1010);
Performing an alignment using luminance masking and / or luminance compensation to generate an alignment image (1022);
And repeating the following steps.
Blurring the high resolution estimated image using a point spread function to generate a blurred image (1014)
Subtracting the blurred image and the alignment image to generate a difference image (1006)
Back projecting the difference image using a point spread inverse function to generate a back projection image (1008)
Performing regularization to generate a smoothed image (1018);
Combining the smoothed image and the backprojected image to generate an enhancement coefficient (1016)
Updating the enhancement coefficient to a high-resolution estimated image and generating a new high-resolution estimated image (1012) The method of claim 1, wherein alignment using luminance masking and / or luminance compensation is performed to generate an alignment image.
Generating a warping vector and a predicted image by performing motion detection (1002);
Performing luminance masking and luminance compensation to generate a luminance mask and a luminance compensation value (1020);
And (1004) generating a registration image by performing registration based on the warping vector, a luminance mask, and a luminance compensation value. The method of claim 2, wherein brightness masking and brightness compensation are performed to generate a brightness mask and a brightness compensation value.
A step of calculating distortion (S402);
Comparing the distortion with a threshold (S404);
If the distortion meets the threshold, marking the reference block as usable for super-resolution and ending the process (S414);
Performing the following steps if the distortion does not meet the threshold.
A step of calculating a luminance difference and generating the luminance compensation value (S406)
Step of performing luminance compensation (S408)
Step of calculating distortion again (S410)
A step of comparing the distortion with a threshold value (S412)
If the distortion satisfies the threshold value and luminance compensation is used, marking the reference block as usable for super-resolution and ending the process (S416)
If the distortion does not satisfy the threshold value, the reference block is marked as unusable for super-resolution, and the process is terminated (S418). The method of claim 2, wherein brightness masking and brightness compensation are performed to generate a brightness mask and a brightness compensation value.
A step of calculating distortion (S502);
Calculating a luminance difference (S504);
Performing luminance compensation (S506);
Calculating luminance compensated distortion (S508);
Comparing the distortion and the luminance compensated distortion with a threshold (S510);
Marking a reference block as unusable for super-resolution when both the distortion and the brightness compensated distortion do not satisfy the threshold, and ending the process (S514);
A method of performing the following steps when the distortion or the luminance compensated distortion satisfies the threshold.
Comparing the distortion with the brightness compensated distortion (S512)
If the distortion is smaller than the luminance compensated distortion, marking the reference block as being usable for super-resolution and ending the process (S516)
When the luminance compensated distortion is smaller than the distortion and luminance compensation is used, a step of marking the reference block as being usable for super-resolution and ending the processing (S518) The method of claim 1, wherein alignment using luminance masking and / or luminance compensation is performed to generate an alignment image.
Performing motion detection to generate a warping vector and a predicted image (S1002);
Performing luminance masking to generate a luminance mask (1020);
And (1004) performing alignment based on the warping vector and a luminance mask to generate an alignment image. The method of claim 5, wherein the luminance mask is generated by performing luminance masking,
A step of calculating distortion (S802);
Comparing the distortion with a threshold value (S804);
Marking the reference block as usable for super-resolution when the distortion satisfies the threshold, and ending the process (S806);
Marking the reference block as being unusable for super-resolution when the distortion does not satisfy the threshold, and ending the processing (S808). The method of claim 1, wherein alignment using luminance masking and / or luminance compensation is performed to generate an alignment image.
Performing motion detection to generate a warping vector and a predicted image (S1002);
Performing luminance compensation to generate a luminance compensation value (1020);
And (1004) performing alignment based on the warping vector and the luminance compensation value to generate an alignment image. The method of claim 7, wherein brightness compensation is performed to generate a brightness compensation value.
Calculating a luminance difference (S902);
Performing luminance compensation. A method comprising the steps of: A method for performing super-resolution processing using luminance masking and luminance compensation on an image sequence,
Generating an initial high-resolution estimated image by performing interpolation (1110);
And repeating the following steps.
Performing warping using luminance masking and / or luminance compensation to generate a warped image sequence (1122)
Blurring the warped image sequence using a point spread function to generate a blurred image sequence (1114)
Subtracting the blurred image sequence from the observed image sequence (input image sequence) to generate a difference image sequence (1106)
Back projecting the difference image sequence using a point spread inverse function to generate a back projection image sequence (1108)
Step of performing regularization to generate a smoothed image (1118)
Combining the smoothed image and the backprojected image sequence to generate an enhancement coefficient (1116)
Updating the enhancement coefficient to a high resolution estimated image to generate a new high resolution estimated image (1112) The method according to claim 9, wherein warping using luminance masking and / or luminance compensation is performed to generate a warped image sequence.
Performing motion detection to generate a warping vector and a predicted image sequence (1102);
Performing luminance masking and luminance compensation to generate a luminance mask and a luminance compensation value (1120);
And (1104) performing warping based on the warping vector, a luminance mask, and a luminance compensation value to generate a warped image sequence. The method of claim 10, wherein brightness masking and brightness compensation are performed to generate a plurality of brightness masks and a plurality of brightness compensation values.
A step of calculating distortion (S402);
Comparing the distortion with a threshold (S404);
If the distortion meets the threshold, marking the reference block as usable for super-resolution and ending the process (S414);
Performing the following steps if the distortion does not meet the threshold.
A step of calculating a luminance difference and generating the luminance compensation value (S406)
Step of performing luminance compensation (S408)
Step of calculating distortion again (S410)
A step of comparing the distortion with a threshold value (S412)
If the distortion satisfies the threshold value and luminance compensation is used, marking the reference block as usable for super-resolution and ending the process (S416)
If the distortion does not satisfy the threshold value, the reference block is marked as unusable for super-resolution, and the process is terminated (S418). The method of claim 10, wherein brightness masking and brightness compensation are performed to generate a plurality of brightness masks and a plurality of brightness compensation values.
A step of calculating distortion (S502);
Calculating a luminance difference (S504);
Performing luminance compensation (S506);
Calculating luminance compensated distortion (S508);
Comparing the distortion and the luminance compensated distortion with a threshold (S510);
Marking a reference block as unusable for super-resolution when both the distortion and the brightness compensated distortion do not satisfy the threshold, and ending the process (S514);
A method of performing the following steps when the distortion or the luminance compensated distortion satisfies the threshold.
Comparing the distortion with the brightness compensated distortion (S512)
If the distortion is smaller than the luminance compensated distortion, marking the reference block as being usable for super-resolution and ending the process (S516)
When the luminance compensated distortion is smaller than the distortion and luminance compensation is used, a step of marking the reference block as being usable for super-resolution and ending the processing (S518) The method according to claim 9, wherein warping using luminance masking and / or luminance compensation is performed to generate a warped image sequence.
Performing motion detection to generate a warping vector and a predicted image sequence (S1002);
Performing luminance masking to generate a luminance mask (1020);
And (1004) performing warping based on the warping vector and the luminance mask to generate a warped image sequence. 14. The method of claim 13, wherein brightness masking is performed to generate a plurality of brightness masks.
A step of calculating distortion (S802);
Comparing the distortion with a threshold value (S804);
Marking the reference block as usable for super-resolution when the distortion satisfies the threshold, and ending the process (S806);
Marking the reference block as being unusable for super-resolution when the distortion does not satisfy the threshold, and ending the processing (S808). The method according to claim 9, wherein warping using luminance masking and / or luminance compensation is performed to generate a warped image sequence.
Performing motion detection to generate a warping vector and a predicted image (S1002);
Performing luminance compensation to generate a luminance compensation value (1020);
And (1004) generating warped images by performing warping based on the warping vector and the luminance compensation value. The method of claim 15, wherein the luminance compensation is performed to generate a plurality of luminance compensation values.
Calculating a luminance difference (S902);
Performing luminance compensation. A method comprising the steps of: In a pre-processing device or a post-processing device, a method for performing super-resolution processing using luminance masking and luminance compensation,
Demultiplexing the input source as necessary to generate an uncompressed image sequence or encoded video bitstream and other bitstreams (602);
Decoding the encoded video bitstream as necessary to generate a decoded image sequence and optionally auxiliary data; (604);
Optionally, performing a pre-processing on the decoded image sequence or the uncompressed image sequence to generate a processed image sequence (606);
Performing the super-resolution processing using luminance masking and luminance compensation according to claim 1 to the processed image sequence to generate a super-resolution-enhanced image sequence (608);
Optionally performing post-processing on the super-resolution enhanced image sequence to generate a processed super-resolution enhanced image sequence (610);
Optionally compressing and optionally storing the processed super-resolution enhanced image sequence (612);
Optionally displaying (614) the processed super-resolution enhanced image sequence. In the encoder, a method of performing super-resolution processing using luminance masking and luminance compensation,
Performing intra prediction (706);
Performing motion detection (702);
Converting (704);
Performing quantization (708);
Performing entropy encoding (710);
Performing inverse quantization (712);
Performing inverse transformation (714);
Performing inverse intra prediction (716);
Performing motion compensation (718);
And (720) performing a super-resolution process using luminance masking and luminance compensation according to claim 1 to a reference image used for the motion detection and motion compensation. The method according to any one of claims 1 to 16, wherein the decoder performs super-resolution processing using luminance masking and luminance compensation.
Performing entropy decoding (740);
Performing inverse quantization (742);
Performing inverse transformation (744);
Performing inverse intra prediction (746);
Performing motion compensation (750);
And a step (748) of performing super-resolution processing using luminance masking and luminance compensation on the reference image used for the motion compensation. In an encoder using inter-layer prediction, a method of performing super-resolution processing using luminance masking and luminance compensation,
Encoding a base layer (1202);
Encoding an enhancement layer using inter-layer prediction (1212);
Performing inter-layer residual prediction between the enhancement layer and the base layer (1204);
Performing inter-layer motion data prediction between the enhancement layer and the base layer (1206);
Performing intra-layer intra prediction between the enhancement layer and the base layer (1208);
The super-resolution processing using luminance masking and luminance compensation according to claim 1-16 for the intra image or decoded image of the base layer used as an additional reference for encoding the enhancement layer. And (1210) performing the method. In a decoder using inter-layer prediction, a method of performing super-resolution processing using luminance masking and luminance compensation,
Decoding the base layer (1202);
Decoding an enhancement layer using inter-layer prediction (1212);
Performing inter-layer residual prediction between the enhancement layer and the base layer (1204);
Performing inter-layer motion data prediction between the enhancement layer and the base layer (1206);
Performing intra-layer intra prediction between the enhancement layer and the base layer (1208);
17. Super-resolution processing using luminance masking and luminance compensation according to claim 1 to an intra image or decoded image of the base layer used as an additional reference for decoding the enhancement layer And (1210) comprising the steps of:

Images