JP5717548B2 - Super-resolution auxiliary information generation device, encoding device, decoding device, and programs thereof - Google Patents

Description

  The present invention uses a super-resolution auxiliary information generation device and a super-resolution auxiliary information generation device that generate super-resolution auxiliary information used when generating a super-resolution image from a reduced image sequence obtained by reducing an original image sequence. The present invention relates to an encoding device, a decoding device that decodes encoded data transmitted from the encoding device, and these programs.

  2. Description of the Related Art In recent years, high definition of imaging devices and display devices has progressed, and a high resolution technology for moving images called super-resolution has been studied (see, for example, Patent Document 1). Ultra-high definition video such as Super Hi-Vision (SHV) called 8K system, or high-definition video like Digital Cinema called 4K system has 4 to 16 times higher resolution than conventional high-definition (HDTV) video. It has come to have.

  Therefore, when super-resolution is performed using two images, after pre-processing such as scene change detection and pixel area mask processing that is not suitable for pixel alignment, super-resolution is performed by pixel reconstruction processing. A technique for creating an image is known (see, for example, Patent Document 2).

  On the other hand, a plurality of image data with different resolutions are hierarchically set for one screen of a certain moving image, and an evaluation value for the motion amount is obtained for the image data with a certain resolution, and image data different from this resolution is obtained. A technique is known in which an evaluation value for the amount of motion is obtained and a final amount of motion is determined from values obtained by adding the evaluation values (see, for example, Patent Document 3).

  In addition, wavelet transformation of a plurality of floors is performed in a spatial direction on one screen of a certain moving image to extract contour information that lowers the priority of the rank including a lot of high-frequency component regions. Divide the screen into blocks, and set the same priority to the block indicated by the contour information so that the priority is higher as the activity (local nature of the image) in the block indicated by the contour information is smaller. A technique is known in which the amount of encoded data is controlled by performing a truncation process in order from the lowest block (see, for example, Patent Document 4).

  The video format of the SHV screen is horizontal 7680 pixels, vertical 4320 lines, time 60 frames / second. Compared with the video format of the HDTV screen, the horizontal and vertical sampling frequencies are relative to the time sampling frequency. Has increased. Therefore, the moving area of the SHV screen shows a larger amount of movement (number of pixels indicating movement in units of frames: pixels / frame) than the HDTV screen when compared with moving images captured at the same angle of view. In the region, it is assumed that the correlation between frames is low and the power in the high frequency region in the time direction is high, the blur amount in the moving region is large, and the power in the high frequency region in the spatial direction is low. . Processing based on these assumptions is effective in estimating the amount of motion between a plurality of frames in encoding processing and super-resolution processing. Note that it is known that the amount of motion of a moving picture of the HDTV standard is generally about several pixels / frame to several tens of pixels / frame.

  In general, a motion vector detection using a small block size (for example, 8 × 8 pixels) is suitable for a pattern with many spatial high-frequency components. On the other hand, since a pattern with few spatial high-frequency components has a high possibility of erroneous motion vector detection at a small block size, motion vector detection using a large block size (for example, 32 × 32 pixels) is effective. Furthermore, in motion vector detection using a large block size, a large motion search range is required because the influence of blur due to large motion can be considered.

  On the other hand, even if the technique of Patent Document 3 is applied to estimate the amount of motion between a plurality of frames in such encoding processing or super-resolution processing, it is not possible to hierarchize based on the power in the spatio-temporal frequency domain. In addition, since processing is performed with a predetermined number of hierarchies, the processing load increases, and it is impossible to detect a motion amount that reflects the influence of blur in the time direction.

  In addition, even if the technique of Patent Document 4 is applied to estimate the amount of motion between a plurality of frames in such encoding processing or super-resolution processing, encoding is performed depending on the rank including a lot of high-frequency component regions. Although the priority of the block for selecting the data amount can be determined, the motion vector cannot be made highly accurate because it is not hierarchized based on the power of the spatio-temporal frequency domain.

  Therefore, the resolution value is determined by detecting the power ratio in the high frequency region in the time direction and / or the power ratio in the low frequency region in the spatial direction, and the block size and motion search range corresponding to the determined resolution value are determined. Hierarchy in which motion vector detection is started from a size hierarchy, and gradually moves to motion vector detection in a hierarchy having a block size smaller than the block size and a size of the motion search range smaller than the size of the motion search range. A technique for performing type motion vector detection is known (see, for example, Patent Document 5).

JP 2010-134582 A JP 2010-134582 A Japanese Patent No. 3334271 Japanese Patent No. 4195978 JP 2011-082700 A

  In estimating the amount of motion between a plurality of frames in encoding processing and super-resolution processing, the technique of Patent Document 5 is applied to reproduce a spatial frequency component close to true when the motion vector detection accuracy is high. be able to. However, when the motion vector detection accuracy is low, there is a problem in that artifacts occur and the image quality deteriorates.

  The present invention has been made in view of the above-described problems, and in addition to motion vector information, super-resolution capable of generating auxiliary information necessary for generating a high-quality super-resolution image. An object is to provide an auxiliary information generation device, an encoding device, a decoding device, and a program thereof.

In order to solve the above problems, a super-resolution auxiliary information generating apparatus according to the present invention generates super-resolution auxiliary information used when generating a super-resolution image from a reduced image sequence obtained by reducing an original image sequence. a resolution supplementary information generating apparatus, the reference frame of the original image sequence, for each block of a predetermined size, a motion vector detecting section for detecting a motion vector between the plurality of reference frames of the original image sequence, the an image reducing unit that generates the reduced image sequence obtained by reducing the original image sequence with a predetermined reduction ratio, generated by the image reducing unit, with respect to the reduced image of the reference frame, using a plurality of first auxiliary information, A super-resolution image is generated for each first auxiliary information, and a super-resolution image having a minimum difference value from an image obtained by reducing the reference frame to the same size as the super-resolution image is defined as a reference frame super-resolution image. Select the reference frame A reference frame super resolution unit for outputting a first auxiliary information used for generating the super-resolution image as a reference frame super-resolution assist information generated by the image reduction unit, a reduced image column of the plurality of reference frames On the other hand, using a plurality of second auxiliary information, a super-resolution image sequence is generated for each second auxiliary information and is output as a reference frame super-resolution image sequence for each second auxiliary information. A complementary image for each of the second auxiliary information by performing pixel interpolation on the reference frame super-resolution image using the reference frame super-resolution image sequence for each of the motion vector and the second auxiliary information. produced, and a plurality of frames reconstructed determination unit difference value is output as the reference frame super-resolution assist information a second auxiliary information used to generate the smallest complementary image of the reference frame, the reference frame Super resolution And a co information and the reference frame super resolution supplementary information, and outputs to said Rukoto as the super-resolution auxiliary information.

Further, in the super-resolution auxiliary information generation device of the present invention, the multiple-frame reconstruction determination unit aligns the reference frame super-resolution image sequence with the motion vector with respect to the reference frame super-resolution image. an alignment processing unit that performs pixel interpolation for generating a positioned image of each of the second auxiliary information by, the positioned image of each of the second auxiliary information, of the original image sequence with the point spread function Pixel complementation is performed so that each frame has the same size, and an image complement processing unit that generates a supplementary image for each second auxiliary information, and a difference value between the complementary image for each second auxiliary information and the reference frame is calculated. And a reference frame super-resolution auxiliary information determining unit that outputs the second auxiliary information used for generating the complementary image that minimizes the difference value as reference frame super-resolution auxiliary information. To.

Further, the super-resolution assist information generating apparatus of the present invention, the reference frame super resolution unit, the reduced image of the reference frame of the decomposition rank n generated by the octave decomposition of the original image sequence by the image reduction unit The reference frame octave decomposition unit that generates the reference frame decomposition image of the high-frequency region components in the horizontal, vertical, and diagonal directions in the decomposition rank n + 1, and the horizontal, vertical, and diagonal generated by the reference frame octave decomposition unit Each of the reference frame decomposition images of the high frequency region component in the direction is set as the spatial low frequency region component, and the image having the zero matrix of the same size set as the spatial high frequency region component is subjected to octave reconstruction processing to obtain horizontal in the decomposition rank n Reference frame octave reconstruction for generating reference frame reconstructed images of high frequency components in vertical, diagonal directions And a filtering process using the first auxiliary information to generate a filtered image on the reference frame reconstructed image, set the filtered image as a high frequency component, and convert the reduced image of the reference frame to a low frequency A reference frame filter applied octave reconstruction unit that performs octave reconstruction processing on an image set as a component to generate a reference frame filter applied reconstructed image for each decomposition order n-1 for each of the first auxiliary information; calculating a low-frequency component of the frame filter application reconstructed image, a difference value between the low-frequency component of the reduced image degradation rank n-1 reference frame generated by the octave decomposition of the original image sequence by the image reduction unit Then, the low-frequency component of the reconstructed image at the resolution rank n−1 that minimizes the difference value is defined as the reference frame super-resolution image. Characterized in that it comprises a reference frame super-resolution image, the super-resolution auxiliary information determining unit.

Further, the super-resolution assist information generating apparatus of the present invention, the reference frame super resolution unit at the image reduction unit of the plurality of reference frames of the decomposition rank n generated by the octave decomposition of the original image sequence A reference frame octave decomposition unit that generates a decomposed image of high-frequency region components in the horizontal, vertical, and oblique directions in the decomposition rank n + 1 by octave decomposition of the reduced image sequence, and horizontal, vertical, and vertical generated by the reference frame octave decomposition unit, Each of the reference frame decomposition images of the high-frequency region component in the diagonal direction is set as a spatial low-frequency region component, and an octave reconstruction process is performed on an image in which a zero matrix of the same size is set as the spatial high-frequency region component. Reference frame octave reconstruction unit that generates a reference frame reconstruction image of each high-frequency region component in the horizontal, vertical, and diagonal directions , Relative to the reference frame reconstructed image, the second to generate a filtered image by performing the filtering process using the auxiliary information, the filtered image set to a high-frequency component of the reduced image column of the plurality of reference frames Low An octave reconstruction process is performed on the image set as the frequency component to generate a reference frame filter applied reconstructed image in the decomposition rank n−1 for each of the second auxiliary information, and a reference for each of the generated second auxiliary information A reference frame filter applied octave reconstruction unit that uses a frame filter applied reconstructed image as the reference frame super-resolution image sequence.

  Furthermore, in the super-resolution auxiliary information generating apparatus according to the present invention, the first auxiliary information and the second auxiliary information are variance values of a Gaussian filter.

Further, the super-resolution assist information generating apparatus of the present invention, the motion vector detecting section, the original time direction over a plurality of frames in the image sequence of the power of the high frequency region percentage is too large for a large block size and a large motion estimation range Hierarchically defined to have a larger block size and larger motion search range as the resolution defined hierarchically to be large or the ratio of power in the low-frequency region in the spatial direction of one frame in a moving image increases. A resolution determination unit that determines a resolution value by detecting a power ratio of the high frequency region in the time direction or a power ratio of the low frequency region in the spatial direction, and the above table. Then, motion vector detection is performed from a hierarchy of block size and motion search range corresponding to the resolution value. A hierarchical motion vector detection unit that starts and gradually shifts to motion vector detection in a layer having a block size smaller than the block size and a size of the motion search range smaller than the size of the motion search range. It is characterized by that.

In order to solve the above-described problem, an encoding apparatus according to the present invention includes the above-described super-resolution auxiliary information generation apparatus, and information on motion vectors detected by the motion vector detection unit, generated by the image reduction unit. reduced image sequence, the reference frame reference frame super resolution supplementary information generated by the super-resolution unit, and coding obtained by encoding the reference frame super-resolution assist information generated by the plurality of frames reconstructed determination unit It is characterized by generating data.

In order to solve the above-described problem, the decoding apparatus according to the present invention provides the motion vector information, the reduced image sequence , the reference frame super-resolution auxiliary information, and the reference frame super-resolution auxiliary from the encoding apparatus. The encoded data of information is acquired, and a super-resolution image of the reduced image sequence is generated.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the above-described super-resolution auxiliary information generation device, encoding device, or decoding device.

  According to the present invention, it is possible to generate auxiliary information necessary for generating a high-resolution super-resolution image separately from the motion vector information. In the present invention, a reference frame super-resolution image obtained by super-resolution processing of the reference frame is generated, and reference frame super-resolution auxiliary information is generated based on the reference frame super-resolution image. Alternatively, it is possible to generate a super-resolution image with higher image quality than when super-resolution is performed using only a plurality of frames. Furthermore, since the auxiliary information is determined so that the difference from the original image is minimized, a high-resolution super-resolution image can be generated.

It is a block diagram which shows the structure of the super-resolution auxiliary information generation device of one Example by this invention. It is a block diagram which shows the structure of the 1st example of the motion vector detection part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a block diagram which shows the structure of the 2nd example of the motion vector detection part in the super-resolution auxiliary information generation apparatus of one Example by this invention. It is a block diagram which shows the structure of the time direction high frequency area | region power calculation part in the super-resolution auxiliary information generation apparatus of one Example by this invention. It is a block diagram which shows the structure of the spatial direction low frequency area | region power calculation part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a figure which illustrates the relationship between the ratio of the power of the low frequency area | region of the spatial direction of the 2nd example of the motion vector detection part in the super-resolution auxiliary information generation apparatus of one Example by this invention, and a motion detection start floor. It is a block diagram which shows the structure of the hierarchical motion detection part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a block diagram which shows the structure of the reference | standard frame super-resolution part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a figure which shows the outline | summary of the process of the reference | standard frame super-resolution part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a figure which shows the image produced | generated by the reference | standard frame super-resolution part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a block diagram which shows the structure of the reference frame super-resolution part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a figure which shows the outline | summary of the process of the reference frame super-resolution part in the super-resolution auxiliary information generation device of one Example by this invention. It is a block diagram which shows the structure of the multi-frame reconstruction determination part in the super-resolution auxiliary information generation device of one Example by this invention. It is a figure which shows the outline | summary of a process of the multi-frame reconstruction determination part in the super-resolution auxiliary | assistant information generation device of one Example by this invention. It is a block diagram which shows the structure of the encoding apparatus of one Example by this invention. It is a block diagram which shows the structure of the super-resolution image production | generation part in the encoding apparatus of one Example by this invention. It is a block diagram which shows the structure of the decoding apparatus of one Example by this invention.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[Super-resolution auxiliary information generator]
FIG. 1 is a block diagram of a super-resolution auxiliary information generating apparatus according to an embodiment of the present invention. The super-resolution auxiliary information generating apparatus 1 includes a motion vector detection unit 10, an image reduction unit 20, a reference frame super-resolution unit 30, a reference frame super-resolution unit 40, and a multiple frame reconstruction determination unit 50. Prepare.

  The motion vector detection unit 10 inputs an original image sequence (frame image sequence), and divides a reference frame of the input original image sequence into a predetermined block area division size. Then, a motion vector is detected between the divided base frame and the reference frame (for example, the previous frame of the base frame and / or the back frame of the base frame), and a plurality of frames are reconstructed from the detected motion vector information. To the unit 50. Details of the motion vector detection unit 10 will be described later with reference to FIGS.

  The image reduction unit 20 receives an original image sequence, reduces the frames of the input original image sequence at a predetermined reduction rate, and converts a reduced image of the reference frame (reference frame reduced image) to a reference frame super-resolution unit. 30 and outputs a reduced image of the reference frame (reference frame reduced image) to the reference frame super-resolution unit 40. In this embodiment, an example in which a reduced image of an original image sequence (frame image sequence) is generated by wavelet decomposition will be described.

The image reduction unit 20 performs wavelet decomposition on the original image F, down-samples, and outputs a low-resolution reference frame reduced image CA _m ⁽ⁿ⁾ . Here, n indicates the decomposition rank, and m indicates the number of frames of the reference frame. In the following description, the subscript m is omitted as appropriate. The wavelet decomposition rank is determined by the size to be reduced.

The wavelet used for the downsampling process is wavelet_n, the horizontal low frequency / vertical low frequency component of decomposition rank n is CA ⁽ⁿ⁾ (χ, ψ), and the horizontal high frequency / vertical low frequency component is CH ⁽ⁿ⁾ ( χ, ψ), horizontal low frequency / vertical high frequency component is CV ⁽ⁿ⁾ (χ, ψ), and horizontal high frequency / vertical high frequency component is CD ⁽ⁿ⁾ (χ, ψ). Note that when the resolution is converted, the sample is halved both vertically and horizontally, and (x, y) in the horizontal and vertical directions is displayed as (χ, ψ). In the following description, (χ, ψ) is omitted as appropriate.

The image reduction unit 20 uses the n-th order discrete wavelet decomposition DWT ⁽ⁿ⁾ (F (x, y, t), wavelet_n) for the original image F (x, y, t) as shown in Expression (1). Disassembled into t indicates time.

[CA ⁽ⁿ⁾ , CH ⁽ⁿ⁾ , CV ⁽ⁿ⁾ , CD ⁽ⁿ⁾ ] = DWT ⁽ⁿ⁾ (F (x, y, t), wavelet_n) (1)

The reference frame super resolving unit 30 performs a super resolving process on the reference frame reduced image CA _m ⁽ⁿ⁾ input from the image reducing unit 20 at a predetermined enlargement ratio, and generates the generated reference frame super resolving image GExCA. _m ⁽ⁿ⁻¹⁾ is output to the multiple frame reconstruction determination unit 50. Details of the reference frame super resolving unit 30 will be described later with reference to FIGS.

The reference frame super resolving unit 40 performs a super resolving process on the reference frame reduced image sequence CA _{m ′} ⁽ⁿ⁾ input from the image reducing unit 20 at a predetermined enlargement rate for each predetermined auxiliary information. The reference frame super-resolution image GExCA _{m ′} ⁽ⁿ⁻¹⁾ for each generated auxiliary information is output to the multiple-frame reconstruction determination unit 50. Here, m ′ indicates the number of reference frames. Thereby, in each reduced image of the reference frame, a plurality of reference frame super-resolution images are obtained for each predetermined auxiliary information. Details of the reference frame super resolving unit 40 will be described later with reference to FIGS. 11 and 12.

The multi-frame reconstruction determination unit 50 performs motion vector information input from the motion vector detection unit 10 for the reference frame super-resolution image GExCA _m ⁽ⁿ⁻¹⁾ input from the reference frame super-resolution unit 30. , And the reference frame super-resolution image GExCA _{m ′} ⁽ⁿ⁻¹⁾ for each auxiliary information input from the reference frame super-resolution unit 40, depending on the block region division size set by the motion vector detection unit 10. Position alignment is performed for each of the divided blocks, and pixel reconstruction of the reference frame super-resolution image GExCA _m ^(n-1) is performed to generate a reconstructed image having the same size as the original image. Then, auxiliary information that minimizes the difference value between the generated reconstructed image and the original image is obtained. Details of the multiple frame reconstruction determination unit 50 will be described later with reference to FIGS. 13 and 14.

[First example of motion vector detection unit]
Next, a first example of the motion vector detection unit 10 in the super-resolution auxiliary information generation device 1 will be described. FIG. 2 is a block diagram showing the configuration of the first example of the motion vector detection unit 10. The motion vector detection unit 10 of the first example includes a block division unit 101 and a block matching processing unit 102.

  The block dividing unit 101 divides the reference frame image into a predetermined block size using the original image at the position of the processing frame in the input original image sequence as a reference frame image, and outputs the divided image to the block matching processing unit 102. The block size may be stored in a storage unit (not shown) and read.

  The block matching processing unit 102 performs block matching on the reference frame image divided from the block input from the block dividing unit 101 using the original image of the reference frame of the reference frame as a reference frame image, and sets the reference frame for each block. And a reference frame are detected, and motion vector information (alignment information) is output. For block matching, for example, an SSD (Sum of Squared Difference) method or an SAD (Sum of Absolute Intensity Difference) method is used. In addition, block matching is performed with decimal pixel accuracy by a process using a parabolic fitting function shown in Equation (2), for example.

  Here, when the pixel position at the search position is x, SSD (x) represents the SSD value at the pixel position, and more specifically, SSD (0) is the center position (position where the SSD value is minimized). SSD (−1) is the SSD value at the position of the adjacent pixel in the −x direction or −y direction from the center position, and SSD (1) is the SSD value at the position of the adjacent pixel in the + x direction or + y direction from the center position. Represents. From equation (2), pixel positions (decimal pixel positions) with decimal pixel precision in the horizontal or vertical direction can be calculated respectively.

[Second example of motion vector detection unit]
Next, a second example of the motion vector detection unit 10 in the super-resolution auxiliary information generation device 1 will be described. FIG. 3 is a block diagram showing the configuration of the second example of the motion vector detection unit 10. The motion vector detection unit 10 of the second example includes a resolution determination unit 16 and a hierarchical motion detection unit 15. Note that image data necessary for processing by each component may be appropriately stored in a storage unit (not shown) included in the motion vector detection unit 10 and read out.

  The resolution determination unit 16 hierarchically defines the resolution so that the larger the ratio of the power in the high frequency region in the time direction over a plurality of frames in the original image is, the larger the block size and the larger the motion search range are, and / or the moving image. The table of resolution defined hierarchically so as to have a larger block size and a larger motion search range as the proportion of power in the low-frequency region in the spatial direction of one frame in FIG. The resolution value is determined by detecting the power ratio of the region and / or the power ratio of the low frequency region in the spatial direction. Specifically, the resolution determination unit 16 includes a time direction high frequency domain power calculation unit 11, a spatial resolution rank determination unit 12, a spatial direction low frequency domain power calculation unit 13, and a motion detection start resolution determination unit 14. .

Temporal high-frequency range power calculation unit 11 includes a reference frame F to be used for reference frame F (t _C) and the motion search to detect the motion vector (t _R), a plurality of frames at time t = t 0 _... t _m F (t ₀ ),..., F (t _C ),..., F (t _R ),..., F (t _m ) are input, and the plurality of pixels for all pixels in the reference frame F (t _C ) After decomposing the frame into the frequency domain of the maximum rank defined in advance in the time direction, the power for each frequency band in the time direction in all pixels is calculated, and the power in the time direction in the time direction in all the calculated pixels is calculated in the time direction. The ratio of the power in the high frequency region is calculated and sent to the spatial resolution rank determining unit 12.

For example, as illustrated in FIG. 4, the time-direction high-frequency domain power calculation unit 11 includes a frame image sequence F at time t = t ₀ ... T _m including a base frame F (t _C ) and a reference frame F (t _R ). (t ₀ ),..., F (t _C ),..., F (t _R ),..., F (t _m ) are input, and Nmax defined in advance in the time direction for all pixels of the reference frame F (t _C ) A time-direction one-dimensional Nmax-order discrete wavelet decomposition processing unit 111 for performing discrete wavelet decomposition on the floor (for example, the fourth floor), and calculating power for each frequency band in the time direction in all pixels of the reference frame F (t _C ), A time-direction frequency band-specific power calculation unit 112 that calculates the ratio of the power in the high-frequency region in the time direction from the calculated power for each frequency band in all the pixels and sends it to the spatial resolution rank determination unit 12. It can be.

  The spatial resolution rank determination unit 12 has a larger dynamic region area as the proportion of power in the high frequency region in the time direction increases from the proportion of power in the high frequency region in the time direction calculated by the time direction high frequency region power calculation unit 11. It is determined that the amount of motion is large, the spatial frequency decomposition rank Ns (that is, the resolution in the spatial direction) is determined according to the power ratio of the high frequency region in the time direction, and information on the determined spatial frequency decomposition rank Ns is stored in the space. Send to the direction low frequency region power calculation unit 13.

  For example, as shown in Table 1, a table defined in advance between the power ratio in the high frequency region in the time direction and the resolution rank Ns of the spatial frequency is held in advance.

The spatial direction low frequency region power calculation unit 13 inputs the base frame F (t _C ) and the reference frame F (t _R ) and information on the spatial frequency decomposition rank Ns, and based on the spatial frequency decomposition rank Ns, A spatial Ns-order discrete wavelet decomposition is performed on the reference frame F (t _C ) and / or the reference frame F (t _R ), power for each spatial frequency band in the frame is calculated, and each calculated spatial frequency band is calculated. The power ratio of the low frequency region in the spatial direction in the frame is calculated from the power, and the calculated power ratio of the low frequency region in the spatial direction and the spatial Ns-order discrete wavelet decomposition data are sent to the motion detection start resolution determination unit 14 To do.

Note that performing the spatial Ns-order discrete wavelet decomposition on both the base frame F (t _C ) and the reference frame F (t _R ) requires a fixed block size and a search range size as later processing. It is advantageous in that hierarchical motion vector detection with variable block size and search range size can be performed on the original image by hierarchically performing wavelet reconstruction when performing hierarchical motion vector detection. In particular, in order to determine resolution for hierarchical motion vector detection, the ratio of the power in the low frequency region in the spatial direction of the reference frame F (t _C ) and the reference frame F (t _R ) It is preferable to select the larger one. In the following description, an example will be described in which spatial Ns-order discrete wavelet decomposition is performed on both the base frame F (t _C ) and the reference frame F (t _R ).

For example, as illustrated in FIG. 5, the spatial direction low frequency region power calculation unit 13 receives the reference frame F (t _C ) and the reference frame F (t _R ), and the spatial frequency determined by the spatial resolution rank determination unit 12. A spatial direction two-dimensional Ns-order discrete wavelet decomposition processing unit that performs spatial Ns-order discrete wavelet decomposition on all the pixels of the base frame F (t _C ) and the reference frame F (t _R ) based on the decomposition rank Ns of and 131, the reference frame F (t _C) and the reference frame F (t _R) of calculating a power of each of the spatial frequency bands in each reference frame F (t _C) from the power of each calculated spatial frequency band and the reference frame The ratio of the power in the low frequency region in the spatial direction in each of F (t _R ) is calculated, and the spatial direction in each of the calculated reference frame F (t _C ) and reference frame F (t _R ) is calculated. The power calculation unit 132 for each spatial frequency band that transmits information having a higher power ratio in the low frequency region in the direction to the motion detection start resolution determination unit 14 can be configured.

  The motion detection start resolution determination unit 14 determines that the power ratio of the low frequency region in the spatial direction is larger from the information on the power ratio of the low frequency region in the spatial direction calculated by the spatial direction low frequency region power calculation unit 13 (space The resolution for starting motion vector detection hierarchically (hereinafter referred to as “motion detection start resolution”) is determined as the dynamic region area is large and the amount of motion is large as the power ratio of the high frequency region in the direction is small. The motion detection start resolution is determined according to the ratio of the power in the low frequency region in the spatial direction, and information on the determined hierarchical motion detection start resolution is obtained. It is sent to the hierarchical motion detector 15. For example, the motion detection start resolution determination unit 14 increases the number of floors at which motion vector detection is started (hereinafter referred to as “motion detection start floor”) ns, as the power ratio in the low frequency region in the spatial direction increases. As described above, the motion detection start floor number ns is determined according to the power ratio of the low frequency region in the spatial direction, and information on the determined motion detection start floor number ns is sent to the hierarchical motion detection unit 15. However, ns ≦ Ns.

  For example, as shown in Table 2, a table defined between the power ratio of the low frequency region in the spatial direction and the motion detection start resolution (or motion detection start floor ns) is held in advance. As the motion detection start floor number ns increases, this means that the original image has a lower resolution, and the block size and the size of the motion search range increase relative to the original image. I mean. Here, for example, the spatial resolution of 1/16 means a reduction in resolution in which 16 pixels are sampled as one pixel at the horizontal sampling frequency Hs and the vertical sampling frequency Vs in the original image.

That is, as shown in FIG. 6, the motion detection start rank ns can be associated with the power ratio of the low frequency region in the spatial direction. For example, when Ns = 4, the respective powers of the low frequency region (LL ⁴ ) and the high frequency region (other than LL ⁴ ) are calculated, and the ratio of the low frequency region (LL ⁴ ) in the whole is 99.5% or more. If so, it is possible to reconstruct an image of only the low-frequency region of the four layers with the motion detection start rank ns = 4 (see FIG. 6D). Similarly, if the ratio of the low frequency region (LL ⁴ ) in the whole is 98.0% or more and less than 99.5%, an image of only three layers of the low frequency region (LL ³ ) can be reconstructed (FIG. 6). (C)), if the ratio of the low frequency region (LL ⁴ ) in the whole is 95.0% or more and less than 98.0%, it is possible to reconstruct an image of only the low frequency region (LL ² ) of two layers. (Refer to FIG. 6B.) If the ratio of the low frequency region (LL ⁴ ) in the whole is less than 95.0%, an image of only one layer of the low frequency region (LL ¹ ) may be reconstructed. (See FIG. 6A).

The hierarchical motion detection unit 15 generates a reference frame F (t _C ) and a reference frame F (t _R ) in the spatial direction corresponding to the motion detection start rank ns in order to generate images of the reference frame F (t _C ) and the reference frame F (t _R ). The spatial ns-order wavelet is reconstructed according to the motion detection start rank ns with respect to the data in the spatial direction two-dimensional Ns-order discrete wavelet decomposition of t _C ) and the reference frame F (t _R ), and a predetermined block size And the motion vector detection is performed with the size of the search range, the spatial ns-1 floor wavelet is reconstructed, and the motion vector detection is performed again with the predetermined block size and the size of the search range. Decrement the floor number and repeat until motion vector detection is performed with the predetermined block size and search range size in the upper hierarchy (ie, the original image level) The In this hierarchical motion vector detection operation, the base frame F (t _C ) and the reference frame F (t _R ) are input, and the block size is sequentially set with respect to the base frame F (t _C ) based on the motion detection start resolution. The processing is similar to the motion vector detection while reducing the size of the search range. However, according to hierarchical motion vector detection by sequentially repeating through spatial ns-order wavelet decomposition and reconstruction, it is not necessary to prepare the block size and the size of the search range that should be sequentially changed according to the hierarchy. Since the motion vector is detected according to the motion detection start rank ns according to the image scene, high accuracy can be expected.

For example, as illustrated in FIG. 7, the hierarchical motion detection unit 15 displays the images of the reference frame F (t _C ) and the reference frame F (t _R ) in the low frequency region in the spatial direction according to the motion detection start rank ns. In order to generate the motion detection unit 151, a spatial ns-order wavelet is reconfigured according to the motion detection start rank ns, and motion vector detection is performed by block matching with decimal pixel accuracy with a predetermined block size and search range size. And the spatial Ns-order discrete wavelet decomposition calculated by the spatial direction low-frequency domain power calculation unit 13 to repeat the motion vector detection process by decrementing the rank until the original image level corresponding to the highest rank is reached. The wavelet reconstruction of the first floor higher in the spatial direction so that the image has a higher floor than the motion detection start floor ns for the data It can be composed of the space the first floor wavelet reconstruction unit 152. sending on the line the motion detector 151. Therefore, the motion detection unit 151 uses the reconstructed images of the reference frame F (t _C ) and the reference frame F (t _R ) obtained from the spatial first-order wavelet reconstruction unit 152 to perform hierarchical motion vector detection processing. The final motion vector can be determined and output repeatedly.

  It is preferable that the motion vector detection is performed only with the highest rank (that is, the first floor) using the block matching method of the decimal pixel position by quadratic function approximation. For example, the decimal pixel position is calculated by parabolic fitting shown in Expression (2).

[Reference frame super-resolution part]
Next, the reference frame super-resolution unit 30 in the super-resolution auxiliary information generating apparatus 1 will be described. FIG. 8 is a block diagram illustrating a configuration of the reference frame super resolving unit 30. The reference frame super-resolution unit 30 includes a reference frame wavelet decomposition unit (reference frame octave decomposition unit) 31, a reference frame wavelet reconstruction unit (reference frame octave reconstruction unit) 32, and a reference frame variance value storage unit 33. A reference frame filter applied wavelet reconstruction unit (reference frame filter applied octave reconstruction unit) 34 and a reference frame super-resolution image / auxiliary information determination unit 35.

FIG. 9 is a diagram showing an outline of the processing of the reference frame super resolving unit 30. FIG. 10 is a diagram illustrating an image generated by the processing of the reference frame super resolving unit 30. Hereinafter, the reference frame super resolving unit 30 will be described with reference to FIGS. 8 to 10. First, the reference frame wavelet decomposition unit 31 obtains a decomposition frame n reference frame reduced image CA _m ⁽ⁿ⁾ (see FIG. 10A) from the image reduction unit 20 (step S11). Then, the reference frame wavelet decomposition unit 31 performs first-order discrete wavelet decomposition on the low-frequency domain component CA _m ⁽ⁿ⁾ of the decomposition rank n, and converts each high-frequency area component in the horizontal, vertical, and diagonal directions in the decomposition rank n + 1. Reference frame decomposed images CH _m ^{(n + 1)} , CV _m ^{(n + 1)} , and CD _m ^{(n + 1)} (see FIG. 10B) are generated (step S12).

The reference frame wavelet reconstruction unit 32 converts each of the reference frame decomposition images CH _m ^{(n + 1)} , CV _m ^{(n + 1)} , and CD _m ^{(n + 1)} of the horizontal, vertical, and diagonal high-frequency region components into spatial low-frequency region components. And a zero matrix of the same size is set as a spatial high-frequency region component (see FIG. 10C) (step S13). Then, the reference frame wavelet reconstruction unit 32 enlarges the horizontal and vertical directions twice with respect to an image in which a zero matrix is set for the spatial high-frequency component by the first-order discrete wavelet reconstruction. Reference frame reconstructed images ExCH _m ⁽ⁿ⁾ , ExCV _m ⁽ⁿ⁾ , and ExCD _m ⁽ⁿ⁾ (see FIG. 10D ^{) of} high frequency region components are generated (step S14). Equation (3) shows an equation obtained by reconstructing the first-order discrete wavelet using CH _m ^{(n + 1)} as a spatial low frequency region component and a zero matrix of the same size as a spatial high frequency region component. 0 indicates a zero matrix.

ExCH ⁽ⁿ⁾ = IDWT ⁽¹⁾ (CH ^{(n + 1)} , 0,0,0, wavelet_n) (3)

The reference frame variance value storage unit 33 stores a plurality of reference frame auxiliary information (first auxiliary information). Specifically, dispersion values (hereinafter simply referred to as dispersion values) σ _CH ⁽ⁿ⁾ , σ _CV ⁽ⁿ⁾ , σ _CD ⁽ⁿ⁾ of a plurality of Gaussian filters, for example, an initial value σ _CH ⁽ⁿ⁾ = Assuming that 0.1, σ _CV ⁽ⁿ⁾ = 0.1, and σ _CD ⁽ⁿ⁾ = 0.1, the data is stored up to 20.1 in 0.5 increments. Alternatively, the variance values σ _CH ⁽ⁿ⁾ , σ _CV ⁽ⁿ⁾ , and σ _CD ⁽ⁿ⁾ may be controlled to change within a predetermined range with an initial value of 0.1. An example of the dispersion value stored in the reference frame dispersion value storage unit 33 is shown in Table 3.

The reference frame filter applied wavelet reconstruction unit 34 acquires the dispersion values σ _CH ⁽ⁿ⁾ , σ _CV ⁽ⁿ⁾ , σ _CD ⁽ⁿ⁾ from the reference frame dispersion value storage unit 33 (step S15). The reference frame filter applied wavelet reconstruction unit 34 then uses the variance values σCH ⁽ⁿ⁾ , σCV ⁽ⁿ⁾ , σCD for the reconstructed images ExCH _m ⁽ⁿ⁾ , ExCV _m ⁽ⁿ⁾ , ExCD _m ⁽ⁿ⁾ . A filtering process using a Gaussian filter using ⁽ⁿ⁾ is performed to generate filtered images GExCH _m ⁽ⁿ⁾ , GExCV _m ⁽ⁿ⁾ , GExCD _m ⁽ⁿ⁾ (see FIG. 10E) (step S16).

Specifically, the reference frame filter applied wavelet reconstruction unit 34 uses the Gaussian filter shown in Equation (4) for the diagonal high frequency region component ExCD _m ⁽ⁿ⁾ to perform the filtering process shown in Equation (5). To generate a filtered image GExCD _m ⁽ⁿ⁾ .

Gauss (x, y, σ _CD ⁽ⁿ⁾ ) = exp (− (x ² + y ² ) / 2 {σ _CD ⁽ⁿ⁾ } ² ) (4)
Here, σ ² represents dispersion.

Further, the reference frame filter applied wavelet reconstruction unit 34 uses the Gaussian filter expanded in the directionality shown in Equation (6) for the vertical high-frequency region component ExCH _m ⁽ⁿ⁾ to perform the filtering shown in Equation (7). Processing is performed to generate a filtered image GExCH _m ⁽ⁿ⁾ .

^{_{ExGauss (x, σ CH (n}} )) = exp (-x 2/2 {σ CH (n)} 2) (6)

Similarly, the reference frame filter applied wavelet reconstruction unit 34 uses the Gaussian filter expanded in the directionality shown in the equation (8) for the horizontal high-frequency region component ExCV _m ⁽ⁿ⁾ , as shown in the equation (9). Processing is performed to generate a filtered image GExCV _m ⁽ⁿ⁾ .

^{_{ExGauss (y, σ CV (n}} )) = exp (-y 2/2 {σ CV (n)} 2) (8)

Next, the reference frame filter applied wavelet reconstruction unit 34 uses the decomposition frame n reference frame reduced image CA _m ⁽ⁿ⁾ input from the image reduction unit 20 as a low frequency component, and the filtered image obtained in step S15. GExCD _m ⁽ⁿ⁾ , GExCH _m ⁽ⁿ⁾ , and GExCV _m ⁽ⁿ⁾ are set as high frequency components (see FIG. 10F) (step S17). Then, the reference frame filter applied wavelet reconstruction unit 34 performs the first-order discrete wavelet reconstruction on the image set in step S17, so that the reference frame filter applied wavelet reconstruction image GExCA _m ^(n-1) (FIG. 10 ( g)) is generated (step S18).

The reference frame super-resolution image / auxiliary information determination unit 35 receives the reference frame filter applied reconstructed image GExCA _m ⁽ⁿ⁻¹⁾ input from the reference frame filter applied wavelet reconstruction unit 34 and the image reduction unit 20. The difference value with respect to the reference frame reduced image CA _m ⁽ⁿ⁻¹⁾ whose decomposition rank is n−1 floor is calculated (step S19). Here, as the difference value, various values such as a sum of absolute values of differences, a product of absolute values of differences, a minimum value of absolute values of differences, or a sum of squares of differences can be used. Further, as shown in the equation (10), a mean square error (RMS) is calculated in the area of the reduced image CA _m ^{(n−1) and} the area of the reconstructed image GExCA _m ⁽ⁿ⁻¹⁾ applied with the reference frame filter. , RMS (CA _m ⁽ⁿ⁻¹⁾ ) and RMS (GExCA _m ⁽ⁿ⁻¹⁾ ) may be obtained, and the difference value may be calculated.

Diff_rms (σCH ⁽ⁿ⁾ , σCV ⁽ⁿ⁾ , σCD ⁽ⁿ⁾ )
= RMS (CA ^(n-1) (σCH ⁽ⁿ⁾ , σCV ⁽ⁿ⁾ , σCD ⁽ⁿ⁾ ))
-RMS (GExCA ^(n-1) (σCH ⁽ⁿ⁾ , σCV ⁽ⁿ⁾ , σCD ⁽ⁿ⁾ ) (10)

The reference frame filter applied wavelet reconstruction unit 34 performs the processing from step S15 to step S18 while changing the values of the variance values σ _CH ⁽ⁿ⁾ , σ _CV ⁽ⁿ⁾ , and σ _CD ⁽ⁿ⁾ , and performs the reference frame super solution. The image / auxiliary information determination unit 35 obtains a dispersion value that minimizes Diff_rms (σCH ⁽ⁿ⁾ , σCV ⁽ⁿ⁾ , σCD ⁽ⁿ⁾ ), and obtains the dispersion value and the reference obtained using the dispersion value The frame filter applied reconstructed image GExCA ^(n-1) is output as a reference frame super-resolution image.

In this embodiment, the dispersion values σ _CH ⁽ⁿ⁾ , σ _CV ⁽ⁿ⁾ , and σ _CD ⁽ⁿ⁾ are determined by comparing GExCA _m ⁽ⁿ⁻¹⁾ and CA _m ^(n−1). By comparing each of CH _m ⁽ⁿ⁾ and GExCH _m ⁽ⁿ⁾ , CV _m ⁽ⁿ⁾ and GExCV _m ⁽ⁿ⁾ , and CD _m ⁽ⁿ⁾ and GExCD _m ⁽ⁿ⁾ , The variance value may be determined.

[Reference frame super-resolution part]
Next, the reference frame super-resolution unit 40 in the super-resolution auxiliary information generating apparatus 1 will be described. FIG. 11 is a block diagram illustrating a configuration of the reference frame super resolving unit 40. The reference frame super resolving unit 40 includes a reference frame wavelet decomposition unit (reference frame octave decomposition unit) 41, a reference frame wavelet reconstruction unit (reference frame octave reconstruction unit) 42, and a reference frame variance value storage unit 43. And a reference frame filter applied wavelet reconstruction unit (reference frame filter applied octave reconstruction unit) 44.

FIG. 12 is a diagram illustrating an outline of processing of the reference frame super resolving unit 40. Hereinafter, the reference frame super resolving unit 40 will be described with reference to FIGS. 10 to 12. First, the reference frame wavelet decomposition unit 41 acquires a reference frame reduced image CA _{m ′} ⁽ⁿ⁾ (see FIG. 10A) having a decomposition rank n from the image reduction unit 20 (step S21). In FIG. 12, the frames before and after the base frame are shown as reference frames, and in this case, m ′ means m ± 1.

The reference frame wavelet decomposition unit 41 performs first-order discrete wavelet decomposition on the reference frame reduced image CA _{m ′} ⁽ⁿ⁾ of decomposition rank n, and refers to the respective high-frequency region components in the horizontal, vertical, and diagonal directions in decomposition rank n + 1. Frame decomposition images CH _{m ′} ^{(n + 1)} , CV _{m ′} ^{(n + 1)} , and CD _{m ′} ^{(n + 1)} (see FIG. 10B) are generated (step S22).

The reference frame wavelet reconstruction unit 42 spatially reduces each of the reference frame decomposition images CH _{m ′} ^{(n + 1)} , CV _{m ′} ^{(n + 1)} , and CD _{m ′} ^{(n + 1)} of the high-frequency region components in the horizontal, vertical, and diagonal directions. A zero matrix of the same size is set as a spatial high frequency domain component as a frequency domain component (see FIG. 10 (c)) (step S23), and doubled horizontally and vertically by first-order discrete wavelet reconstruction and referenced. Frame reconstructed images ExCH _{m ′} ⁽ⁿ⁾ , ExCV _{m ′} ⁽ⁿ⁾ , and ExCD _{m ′} ⁽ⁿ⁾ (see FIG. 10D) are generated (step S24).

The reference frame variance value storage unit 43 stores a plurality of reference frame auxiliary information (second auxiliary information). Specifically, similar to the reference frame variance value storage unit 33, the variance values σ _CH ⁽ⁿ⁾ , σ _CV ⁽ⁿ⁾ , σ _CD ⁽ⁿ⁾ of a plurality of Gaussian filters as exemplified in Table 3 are stored. To do.

The reference frame filter applied wavelet reconstruction unit 44 acquires the dispersion values σ _CH ⁽ⁿ⁾ , σ _CV ⁽ⁿ⁾ , σ _CD ⁽ⁿ⁾ from the reference frame dispersion value storage unit 43 (step S25), and regenerates the reference frame. Filtering processing by a Gaussian filter using variance values σCH ⁽ⁿ⁾ , σCV ⁽ⁿ⁾ , σCD ⁽ⁿ⁾ for the constituent images ExCH _{m ′} ⁽ⁿ⁾ , ExCV _{m ′} ⁽ⁿ⁾ , ExCD _{m ′} ⁽ⁿ⁾ To generate filtered images GExCH _{m ′} ⁽ⁿ⁾ , GExCV _{m ′} ⁽ⁿ⁾ , GExCD _{m ′} ⁽ⁿ⁾ (see FIG. 10E) (step S26). A specific example is the same as that in step S16, and is omitted.

Subsequently, the reference frame filter applied wavelet reconstruction unit 44 uses the decomposition frame n reference frame reduced image CA _{m ′} ⁽ⁿ⁾ input from the image reduction unit 20 as a low frequency component, and performs the filtering obtained in step S15. The images GExCD _{m ′} ⁽ⁿ⁾ , GExCH _{m ′} ⁽ⁿ⁾ , and GExCV _{m ′} ⁽ⁿ⁾ are set as high frequency components (see FIG. 10F) (step S27). Then, the reference frame filter applied wavelet reconstruction unit 44 performs first-order discrete wavelet reconstruction on the image set in step S16, and the reference frame filter applied wavelet reconstruction image GExCA _{m ′} ⁽ⁿ⁻¹⁾ (FIG. 10 ⁾ . (See (g)) is generated (step S28). Here, the reference frame filter applied wavelet reconstruction unit 34 of the reference frame super-resolution unit 30 has a variance that minimizes the difference value from the reference frame reduced image CA _m ⁽ⁿ⁻¹⁾ of the decomposition rank n−1. One reference frame super-resolution image GExCA ⁽ⁿ⁻¹⁾ obtained using the value is output, but the reference frame filter applied wavelet reconstruction unit 44 stores the auxiliary information stored in the reference frame variance value storage unit 43. A plurality of reference frame filter applied reconstructed images GExCA _{m ′} ⁽ⁿ⁻¹⁾ for each (dispersion value) are output as a reference frame super-resolution image sequence.

[Multi-frame reconstruction determination unit]
Next, the multi-frame reconstruction determination unit 50 in the super-resolution auxiliary information generation device 1 will be described. FIG. 13 is a block diagram illustrating a configuration of the multiple frame reconstruction determination unit 50. The multi-frame reconstruction determination unit 50 includes an alignment processing unit 51, an image complement processing unit 52, and a reference frame super-resolution auxiliary information determination unit 53. FIG. 14 is a diagram illustrating an outline of processing of the multiple frame reconstruction determination unit 50. Hereinafter, the multiple frame reconstruction determination unit 50 will be described with reference to FIGS. 13 and 14.

  The alignment processing unit 51 receives the motion vector information input from the motion vector detection unit 10 and the reference frame super resolution unit 40 from the reference frame super resolution image input from the reference frame super resolution unit 30. Registration (registration) is performed using the input reference frame super-resolution image sequence (step S31). That is, pixel interpolation is performed by aligning the image between pixels of the base frame super-resolution image with the reference frame super-resolution image using the motion vector. By complementing the image between pixels of the reference frame super-resolution image, the reference frame super-resolution image can be enlarged. However, since there are cases where the image between all the pixels cannot be complemented (for example, a still region), the image complementation processing unit 52 further performs complementation processing.

  The image complement processing unit 52 complements the image after registration for each auxiliary information input from the registration processing unit 51 so as to be the same size as the original image using a point spread function, and for each auxiliary information. The complementary image is output to the reference frame super-resolution auxiliary information determination unit 53. A conventional MAP method can be used as a method for performing pixel interpolation using a point spread function. By performing pixel complementation using a point spread function, it is possible to perform pixel complementation even for a still region.

In the conventional MAP method, the posterior probability p (x | y _{1 when} x is a frame image vector obtained by restoring a high resolution image, y is a super resolution original frame image vector having a low resolution, and N is the number of input frames. , Y ₂ ,..., Y _N ) is a method for estimating a high-resolution frame image.

  According to the conventional MAP method, when the error distribution is assumed to be a normal distribution, the estimated high-resolution frame image x (^) is calculated by Expression (11).

また、式（１１）の点広がり関数ＰＳＦのＢ_ｋを、２次元ガウス関数近似による点広がり関数とし、位置合わせ情報（つまり、フレーム間動きベクトルの向きと大きさに対応する情報）に応じて、２次元ガウス関数近似による点広がり関数の拡張を行うようにしてもよい。この拡張されたＰＳＦ関数を動き量適応型点広がり関数カーネルＥ・Ｂ_ｋとする。

Further, B _k of the point spread function PSF of the equation (11) is a point spread function by two-dimensional Gaussian function approximation, and according to the alignment information (that is, information corresponding to the direction and magnitude of the inter-frame motion vector). You may make it extend the point spread function by two-dimensional Gaussian function approximation. This expanded PSF function is defined as a motion amount adaptive type point spread function kernel E · B _k .

Formula (12) shows a MAP reconstruction processing formula using the motion amount adaptive type point spread function kernel E · B _k .

より具体的に、「２次元ガウス関数近似による点広がり関数ＰＳＦ」と、「動き量適応型点広がり関数（拡張された２次元ガウス関数近似による点広がり関数ＰＳＦ）」について、説明する。ＰＳＦを、２次元のガウス関数で近似するにあたり、式（１３）のように、半値幅をｗとし、面積が１の規格化ガウス関数の振幅をαとして定義する。

More specifically, the “point spread function PSF based on two-dimensional Gaussian function approximation” and the “motion amount adaptive point spread function (point spread function PSF based on extended two-dimensional Gaussian function approximation)” will be described. In approximating the PSF with a two-dimensional Gaussian function, the half-value width is defined as w and the amplitude of the normalized Gaussian function with an area of 1 is defined as α as shown in Equation (13).

From Equation (12), the two-dimensional Gaussian function is as shown in Equation (14). Here, the horizontal reference position is x ₀ , and the vertical reference position is y ₀ .

位置合わせ情報ｍの水平方向の動き量をｍ_ｘ、垂直方向の動き量をｍ_ｙとすると、式（１３）のガウス関数における水平方向と垂直方向の半値幅は、式（１５）のように制御される。

Horizontal movement amount m _x alignment information _m, the movement amount in the vertical direction and m _y, the half-width of the horizontal and vertical directions in a Gaussian function of Equation (13), as in equation (15) Be controlled.

式（１５）は、位置合わせ情報の向きと大きさに応じて点広がり関数の広がり方向と広がり量を制御するものである。画像補完処理部５２は、式（１２）のカーネルＥ・Ｂ_ｋを式（１５）の２次元のガウス関数として、画像拡大倍率に応じて式（１５）の半値幅ｗを決定し、式（１４）のλを変化させながら式（１４）の繰り返し演算を行い、最適な補完画像を求めることができる。

Expression (15) controls the spread direction and spread amount of the point spread function according to the direction and size of the alignment information. The image complement processing unit 52 determines the half-value width w of the equation (15) according to the image enlargement magnification using the kernel E · B _k of the equation (12) as a two-dimensional Gaussian function of the equation (15). The optimal complementary image can be obtained by repeating the calculation of Expression (14) while changing λ of 14).

  The reference frame super-resolution auxiliary information determination unit 53 calculates a difference value between the complementary image and the original image for each auxiliary information input from the image complement processing unit 52 for each block. Then, among the complementary images for each auxiliary information, the complementary image having the smallest difference from the original image is selected for each block, and the auxiliary information used for generating the complementary image is determined as the reference frame super-resolution auxiliary information. And output to the outside (step S31). It is possible to determine the reference frame super-resolution auxiliary information so that the difference value between the complementary image and the image obtained by simply enlarging the reduced image is minimized, but by making the original image a comparison target, A high-resolution super-resolution image close to the original image can be generated. Here, as the difference value, various values such as a sum of absolute values of differences, a product of absolute values of differences, a minimum value of absolute values of differences, or a sum of squares of differences can be used. Further, a mean square error (RMS) between the complementary image and the original image may be obtained for each block, and a difference value between the two may be calculated.

  It should be noted that a computer can be suitably used to function as the above-described super-resolution auxiliary information generating device 1, and such a computer describes the processing contents for realizing each function of the super-resolution auxiliary information generating device 1. The program can be realized by storing the program in a storage unit of the computer and reading and executing the program by a CPU (central processing unit) of the computer.

  As described above, according to the super-resolution auxiliary information generating apparatus 1 and the program thereof, it is possible to generate the super-resolution auxiliary information necessary for generating a high-resolution super-resolution image. In addition, by performing pixel interpolation using the point spread function in the multiple frame reconstruction determination unit 50, it is possible to generate a super-resolution image for a still region.

[Encoding device]
Next, an encoding apparatus using the super-resolution auxiliary information generation apparatus 1 described above will be described. FIG. 15 is a block diagram showing a configuration of an encoding apparatus according to an embodiment of the present invention. The encoding device 60 includes a super-resolution auxiliary information generation device 1, a subtraction unit 61, a quantization unit 62, a variable-length encoding unit 63, an inverse quantization unit 64, and a super-resolution image generation unit 65. , An adder 66, a frame memory 67, and a motion compensator 68.

  The subtraction unit 61 generates a difference image between the original image and the predicted image input from the motion compensation unit 68 and outputs the difference image to the super-resolution auxiliary information generation device 1.

When the super-resolution auxiliary information generating device 1 is applied to the encoding device 60, the super-resolution auxiliary information generating device 1 is configured to use motion vector information, base frame super-resolution auxiliary information, and reference frame super-resolution auxiliary information. In addition, the reference frame reduced image CA _m ⁽ⁿ⁾ generated by the image reducing unit 20 is also output to the outside. The super-resolution auxiliary information generation device 1 performs the above-described super-resolution auxiliary information generation processing on the difference image input from the subtraction unit 61, outputs the reference frame reduced image to the quantization unit 62, and the motion vector. The information is output to the variable length encoding unit 63 and the motion compensation unit 68, and the motion vector information, the reference frame super-resolution auxiliary information, and the reference frame super-resolution auxiliary information are output to the variable-length encoding unit 63.

  The quantization unit 62 performs a quantization process on the reference frame reduced image input from the super-resolution auxiliary information generating apparatus 1 and outputs the quantization process to the variable length encoding unit 63 and the inverse quantization unit 64.

  The variable length coding unit 63 includes quantized data of the reference frame reduced image input from the quantization unit 62, motion vector information input from the super-resolution auxiliary information generating device 1, reference frame super-resolution auxiliary information, Then, variable length encoding processing is performed on the reference frame super-resolution auxiliary information, and a bit stream of encoded data is generated and output to the decoding side.

  The inverse quantization unit 64 performs inverse quantization processing on the quantized data of the reference frame reduced image input from the quantization unit 62 and outputs the result to the super-resolution image generation unit 65.

  The super-resolution image generation unit 65 uses the reference frame reduced image, the motion vector information, the reference frame super-resolution auxiliary information, and the reference frame super-resolution auxiliary information that are input from the super-resolution auxiliary information generation device 1. A resolution image is generated and output to the adder 66.

  FIG. 16 is a block diagram illustrating a configuration of the super-resolution image generation unit 65. The super-resolution image generation unit 65 includes a super-resolution unit 651 and a multi-frame reconstruction unit 652. The super-resolution unit 651 includes a reference frame wavelet decomposition unit 6511, a reference frame wavelet reconstruction unit 6512, a reference frame filter applied wavelet reconstruction unit 6513, a memory 6514, a reference frame wavelet decomposition unit 6515, and a reference frame wavelet. A reconstruction unit 6516 and a reference frame filter applied wavelet reconstruction unit 6517 are provided. The multiple frame reconstruction unit 652 includes an alignment processing unit 6521 and an image complement processing unit 6522.

  The processes of the reference frame wavelet decomposition unit 6511, the reference frame wavelet reconstruction unit 6512, and the reference frame filter applied wavelet reconstruction unit 6513 are the same as the reference frame wavelet decomposition unit 31 and the reference frame wavelet reconstruction unit of the reference frame super-resolution unit 30. 32 and the processing of the reference frame filter applied wavelet reconstruction unit 33. However, the reference frame filter applied wavelet reconstruction unit 6513 performs a filtering process using the input reference frame super-resolution auxiliary information.

  The memory 6514 stores the reference frame reduced image. The processes of the reference frame wavelet decomposition unit 6515, the reference frame wavelet reconstruction unit 6516, and the reference frame filter applied wavelet reconstruction unit 6517 are the reference frame wavelet decomposition unit 41 of the reference frame super-resolution unit 40, the reference frame wavelet reconstruction. This is the same as the processing of the unit 42 and the reference frame filter applied wavelet reconstruction unit 43. However, the reference frame wavelet decomposition unit 6515 acquires a reference frame reduced image from the memory 6514, and the reference frame filter applied wavelet reconstruction unit 6517 performs a filtering process using the input reference frame super-resolution auxiliary information.

  Similar to the alignment processing unit 51, the alignment processing unit 6521 applies input motion vector information and reference filter application to the reference frame super-resolution image input from the reference frame filter applied wavelet reconstruction unit 6513. Registration (registration) is performed using the reference frame super-resolution image input from the wavelet reconstruction unit 6517. Similar to the image complement processing unit 52, the image complement processing unit 6522 performs pixel complementation on the image after registration input from the registration processing unit 6521 using a point spread function, and a reference frame having the same size as the original image A super-resolution image is generated.

  The addition unit 66 adds the reference frame super-resolution image input from the super-resolution image generation unit 65 and the predicted image obtained from the motion compensation unit 68 to generate a decoded image, and outputs the decoded image to the frame memory 67. To do.

  The motion compensation unit 68 performs motion compensation on the reference frame super-resolution image stored in the frame memory 67 using the motion vector information input from the super-resolution auxiliary information generation device 1 to generate a prediction image. The predicted image is output to the subtraction unit 61.

  Note that a computer can be suitably used to cause the above-described encoding device 60 to function, and such a computer stores a program describing processing contents for realizing each function of the encoding device 60 in the storage of the computer. The program can be realized by reading out and executing the program by the CPU of the computer.

[Decoding device]
Next, a decoding device using the super-resolution auxiliary information generation device 1 described above will be described. FIG. 17 is a block diagram showing a configuration of a decoding apparatus according to an embodiment of the present invention. The decoding device 70 includes a variable length decoding unit 71, an inverse quantization unit 72, a super-resolution image generation unit 73, an addition unit 74, a frame memory 75, and a motion compensation unit 76.

  The variable length decoding unit 71 performs variable length decoding processing on the encoded data input from the encoding device 60, outputs the quantized data of the reference frame reduced image to the inverse quantization unit 72, and outputs the reference frame super solution. The image auxiliary information and the reference frame super-resolution auxiliary information are output to the super-resolution image generation unit 73, and the motion vector information is output to the super-resolution image generation unit 73 and the motion compensation unit 76.

  The inverse quantization unit 72 performs inverse quantization processing on the quantized data of the reference frame reduced image input from the variable length decoding unit 71 and outputs the result to the super-resolution image generation unit 73.

  The super-resolution image generation unit 73 includes a reference frame reduced image input from the inverse quantization unit 72, motion vector information input from the variable length decoding unit 71, reference frame super-resolution auxiliary information, and reference frame super-resolution. A reference frame super-resolution image is generated using the image auxiliary information and output to the adder 74. The detailed configuration of the super-resolution image generation unit 73 is the same as that of the super-resolution image generation unit 65 shown in FIG.

  The adder 74 adds the reference frame super-resolution image input from the super-resolution image generator 73 and the predicted image input from the motion compensator 76 to restore the image, and stores the decoded image in the frame memory 75. And output to the outside.

  The motion compensation unit 76 performs motion compensation on the reference frame super-resolution image stored in the frame memory 75 using the motion vector information input from the variable length decoding unit 71 to generate a predicted image, and the predicted image Is output to the adder 74.

  Note that a computer can be suitably used to cause the above-described decryption apparatus 70 to function, and such a computer stores a program describing processing contents for realizing each function of the decryption apparatus 70 in a storage unit of the computer. This can be realized by storing the program and executing it by the CPU of the computer.

  Thus, according to the encoding device 60 and the program thereof, the reduced image of the reference frame and the super-resolution auxiliary information generated by the super-resolution auxiliary information generating device 1 can be encoded and transmitted. In addition, according to the decoding device 70 and its program, a high-resolution super-resolution image can be decoded from the reduced image of the reference frame and the super-resolution auxiliary information.

  Although the above-described embodiments have been described as representative examples, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes may be made without departing from the scope of the claims. Is possible. For example, in the embodiment, downsampling is performed by wavelet decomposition, but it is not limited to wavelet decomposition as long as octave decomposition is performed.

  The present invention is useful for generating a super-resolution image, for example, an ultra-high definition video such as Super Hi-Vision (SHV).

DESCRIPTION OF SYMBOLS 1 Super-resolution auxiliary | assistant information generation apparatus 10 Motion vector detection part 11 Time direction high frequency domain power calculation part 12 Spatial decomposition rank determination part 13 Spatial direction low frequency area power calculation part 14 Motion detection start resolution determination part 15 Hierarchical motion detection part 16 Resolution determination unit 20 Image reduction unit 30 Reference frame super-resolution unit 31, 6511 Reference frame wavelet decomposition unit 32, 6512 Reference frame wavelet reconstruction unit 33 Reference frame variance value storage unit 34, 6513 Reference frame filter applied wavelet reconstruction unit 35 Reference frame super-resolution image / auxiliary information determination unit 40 Reference frame super-resolution unit 41, 6515 Reference frame wavelet decomposition unit 42, 6516 Reference frame wavelet reconstruction unit 43 Reference frame variance value storage unit 44, 6517 Reference frame frame Filtering wavelet reconstruction unit 50 Multiple frame reconstruction determination unit 51, 6521 Registration processing unit 52, 6522 Image interpolation processing unit 53 Reference frame super-resolution auxiliary information determination unit 61 Subtraction unit 62 Quantization unit 63 Variable length coding unit 64, 72 Inverse quantization unit 65, 73 Super-resolution image generation unit 66, 74 Addition unit 67, 75 Frame memory 68, 76 Motion compensation unit 71 Variable length decoding unit 101 Block division unit 102 Block matching processing unit 111 Time direction 1 Dimensional Nmax-order discrete wavelet decomposition processing unit 112 Power calculation unit by time direction frequency band 131 Spatial direction two-dimensional Ns-order discrete wavelet decomposition processing unit 132 Power calculation unit by spatial direction frequency band 151 Motion detection unit 152 Spatial first-order wavelet reconstruction unit 651 Super-resolution part 652 Multiple frames Reconstruction unit

Claims (11)

A super-resolution auxiliary information generating device that generates super-resolution auxiliary information used when generating a super-resolution image from a reduced image sequence obtained by reducing an original image sequence,
The reference frame of the original image sequence, for each block of a predetermined size, a motion vector detecting section for detecting a motion vector between the plurality of reference frames of the original image sequence,
An image reducing unit that generates the reduced image sequence obtained by reducing the original image sequence with a predetermined reduction ratio,
Said it generated by the image reduction unit, with respect to the reduced image of the reference frame, using a plurality of first auxiliary information, each first auxiliary information to generate a super-resolution image, ultra resolution the reference frame The super-resolution image having the smallest difference value between the image and the image reduced to the same size is selected as the reference frame super-resolution image, and the first auxiliary information used for generating the reference frame super-resolution image is used as the reference frame. A reference frame super-resolution part to be output as super-resolution auxiliary information;
The image reducing unit is generated by, with respect to the reduced image sequence of the plurality of reference frames by using a plurality of second auxiliary information for each second auxiliary information to generate a super-resolution image sequence, the second auxiliary A reference frame super-resolution unit that outputs a reference frame super-resolution image sequence for each information ;
For the base frame super-resolution image, pixel interpolation is performed using the motion vector and a reference frame super-resolution image sequence for each second auxiliary information to generate a complementary image for each second auxiliary information, and a plurality of frames reconstructed determination unit difference value is output as the reference frame super-resolution assist information a second auxiliary information used to generate the smallest complementary image of the reference frame,
It said reference frame and super-resolution assist information, and the reference frame super resolution supplementary information, the super-resolution assist information generating apparatus according to claim also be output from the super-resolution auxiliary information. The multi-frame reconstruction determination unit
Alignment for generating a registration image for each of the second auxiliary information by performing pixel interpolation by aligning the reference frame super-resolution image sequence with the motion vector with respect to the reference frame super-resolution image A processing unit;
Alignment image for each of the second auxiliary information, an image complementary to the aforementioned pixel interpolation so that the frame and the same size of the original image sequence with the point spread function to generate a complementary image of each of the second auxiliary information A processing unit;
The difference value between the complementary image for each of the second auxiliary information and the reference frame is calculated, and the second auxiliary information used to generate the complementary image that minimizes the difference value is output as reference frame super-resolution auxiliary information. A reference frame super-resolution auxiliary information determination unit;
The super-resolution auxiliary information generation device according to claim 1, comprising: The reference frame super-resolution part is
Wherein a reduced image of the reference frame of the decomposition rank n generated by the octave decomposition of the original image sequence by the image reduction unit and octave exploded, horizontal in the degradation rank n + 1, the vertical reference frame in an oblique direction of the high-frequency domain components A reference frame octave decomposition unit for generating a decomposition image;
Each of the reference frame decomposition images of the horizontal, vertical, and diagonal high frequency region components generated by the reference frame octave decomposition unit is a spatial low frequency region component, and an image in which a zero matrix of the same size is set as the spatial high frequency region component A reference frame octave reconstruction unit that performs octave reconstruction processing on the decomposition rank n to generate a reference frame reconstructed image of each high-frequency region component in the horizontal, vertical, and diagonal directions;
The reference frame reconstructed image is filtered using the first auxiliary information to generate a filtered image, the filtered image is set as a high frequency component, and the reduced image of the reference frame is set as a low frequency component A reference frame filter applied octave reconstruction unit that performs an octave reconstruction process on the generated image and generates a reference frame filter applied reconstructed image in decomposition rank n-1 for each of the first auxiliary information;
Wherein the low-frequency component of the reference frame filter application reconstructed image, a difference value between the low-frequency component of the reduced image degradation rank n-1 reference frame generated by the octave decomposition of the original image sequence by the image reduction unit A reference frame super-resolution image / super-resolution auxiliary information determination unit that uses the low-frequency component of the reconstructed image in the decomposition rank n−1 that minimizes the difference value as the reference frame super-resolution image,
The super-resolution auxiliary information generation device according to claim 1, comprising: The reference frame super-resolution unit is
Wherein the plurality of reduced images sequence of the reference frame by octave exploded, horizontal in the degradation rank n + 1, vertical, diagonal direction high-frequency domain components of the decomposition rank n generated by the octave decomposition of the original image sequence by the image reduction unit A reference frame octave decomposition unit for generating a decomposition image of
Each of the horizontal, vertical, and diagonal high-frequency region component reference frame decomposition images generated by the reference frame octave decomposition unit is a spatial low-frequency region component, and an image in which a zero matrix of the same size is set as the spatial high-frequency region component A reference frame octave reconstruction unit that performs octave reconstruction processing on the decomposition rank n to generate reference frame reconstructed images of horizontal, vertical, and diagonal high-frequency region components;
The reference frame reconstructed image is subjected to a filtering process using the second auxiliary information to generate a filtered image, the filtered image is set as a high frequency component, and the reduced image sequence of the plurality of reference frames is set to a low frequency An octave reconstruction process is performed on the image set as the component to generate a reference frame filter applied reconstructed image in decomposition rank n−1 for each of the second auxiliary information, and a reference frame for each of the generated second auxiliary information A reference frame filter application octave reconstruction unit that uses the filter application reconstructed image as the reference frame super-resolution image sequence;
The super-resolution auxiliary information generation device according to claim 1, comprising:   5. The super-resolution auxiliary information generation device according to claim 1, wherein the first auxiliary information and the second auxiliary information are variance values of a Gaussian filter. The motion vector detection unit
Spatial direction of a frame in said hierarchically defining the resolution so that the size of a large block size and a large motion estimation range ratio of power in the time direction of the high-frequency region over a plurality of frames is large in the original image sequence, or a moving image A table of resolutions defined hierarchically so that the larger the power ratio in the low frequency region is, the larger the block size and the larger the motion search range is,
A resolution determination unit that determines a resolution value by detecting a power ratio of the high-frequency region in the time direction or a power ratio of the low-frequency region in the spatial direction;
With reference to the table, motion vector detection is started from the hierarchy of the block size and the motion search range corresponding to the resolution value, and the block size and the motion search range are gradually smaller than the block size. A hierarchical motion vector detection unit that shifts to motion vector detection in a hierarchy of a size of a motion search range smaller than
The super-resolution auxiliary information generation device according to claim 1, comprising: The super-resolution auxiliary information generating device according to any one of claims 1 to 6,
Information on the motion vector detected by the motion vector detection unit, a reduced image sequence generated by the image reduction unit, reference frame super-resolution auxiliary information generated by the reference frame super-resolution unit, An encoding apparatus that generates encoded data obtained by encoding reference frame super-resolution auxiliary information generated by a configuration determination unit. The encoded data of the motion vector information, the reduced image sequence , the reference frame super-resolution auxiliary information, and the reference frame super-resolution auxiliary information is acquired from the encoding device according to claim 7, and the reduced image is obtained. A decoding apparatus that generates a super-resolution image of a column .   A super-resolution auxiliary information generation program for causing a computer to function as the super-resolution auxiliary information generation device according to any one of claims 1 to 6.   An encoding program for causing a computer to function as the encoding device according to claim 7.   A decoding program for causing a computer to function as the decoding device according to claim 8.

Images