JP5498970B2 - Image reduction device, image enlargement device, and program thereof - Google Patents

Description

  The present invention relates to a technique for performing image reduction on an original image on a transmission side and transmitting the image, and performing image enlargement on a reception side to restore the image, and in particular, reducing and reducing an original image sequence composed of a plurality of original image frames The present invention relates to an image reduction device for generating an image sequence, an image enlargement device for enlarging and restoring a reduced image sequence, and a program thereof.

  2. Description of the Related Art In recent years, high-definition imaging devices and display devices have been advanced, and a moving image high-resolution technique called super-resolution has been studied (for example, see Patent Document 1). Ultra-high definition video such as Super Hi-Vision (SHV) called 8K system, or high-definition video like Digital Cinema called 4K system has a resolution that is 4 to 16 times higher than that of conventional Hi-Vision (HV) video. It has come to have.

  Therefore, when performing super-resolution using two images, after pre-processing such as scene change detection and pixel area mask processing that is not suitable for pixel alignment, super-resolution images are obtained by pixel reconstruction processing. Is known (see, for example, Patent Document 2).

  On the other hand, a technique is known in which an original image is reduced (reduced in resolution) and transmitted on the transmitting side, and a restoration is performed on the receiving side by enlarging (higher resolution). When performing, as auxiliary information when creating a high-resolution image on the receiving side, it is instructed to create a high-resolution image at a location where the color tone of the reduced image and the original original image is similar using color tone There is a technique for generating auxiliary information (see, for example, Patent Document 3).

  When displaying an image on a display device with a low resolution, the image is divided into blocks, the spatial high frequency components of each block are detected, the representative value for each block is determined, the image is reduced, and the previously detected space A technique is known in which a high-frequency component is used as auxiliary information for edge region display, for example (see, for example, Patent Document 4).

  In addition, the technology that optimally encodes the low-resolution image in the technology that performs image reduction (reduction in resolution) on the transmission side for transmission and transmission and enlargement (higher resolution) on the reception side for restoration. There is known a technique for generating an encoding parameter used for generating ancillary information, and auxiliary information that has been scale-converted in order to apply the encoding parameter to a high-resolution image (see, for example, Patent Document 5).

JP 2009-105490 A JP 2010-134582 A Japanese Patent No. 2542726 Japanese Patent No. 4457789 Japanese Patent No. 4523522

  As in the prior art, the auxiliary information is used for image super-resolution, encoding, edge display, and the like using the spatial high-frequency component of the original image as a parameter. Therefore, also when generating a reduced image sequence by reducing an original image sequence composed of a plurality of original image frames, using such auxiliary information is a process for executing image enlargement on the reduced image sequence (hereinafter, referred to as “reduced image sequence”). This is considered effective for “multi-frame super-resolution processing”.

  In general, in the multi-frame super-resolution processing, such auxiliary information is not used, and alignment (for example, motion compensation based on motion estimation) from one processed frame in the reduced image sequence to its preceding and subsequent frames is performed. Perform pixel enlargement with pixel interpolation. Therefore, in the multi-frame super-resolution processing that does not use auxiliary information, since alignment is performed using only the reduced image sequence, there is a problem that alignment accuracy is low.

  Therefore, it is conceivable that highly accurate alignment information can be acquired by performing alignment on the original image sequence, and this alignment information can be used as auxiliary information. In this case, at the time of multi-frame super-resolution processing, image enlargement is performed through a restoration process using a predetermined point spread function. However, since this alignment information does not consider the restoration process by the predetermined point spread function, there is room for further improvement in the image quality after the multi-frame super-resolution processing.

  The present invention has been made in view of the above-described problems, and an image reduction device that reduces an original image sequence composed of a plurality of original image frames to generate a reduced image sequence, and enlarges and restores the reduced image sequence. An object of the present invention is to provide an image enlargement apparatus and a program thereof.

  The image reduction apparatus of the present invention is an apparatus that generates a reduced image sequence by reducing an original image sequence composed of a plurality of original image frames, and generates auxiliary information used for a multi-frame super-resolution process. In the conventional multi-frame super-resolution processing that does not use conventional auxiliary information, alignment is performed using a reduced image sequence, whereas the image reduction device of the present invention uses an original image sequence. This method is used to generate registration information with higher accuracy than that generated by using a reduced image sequence, and in consideration of the restoration process using a point spread function during multi-frame super-resolution. The alignment information is corrected, and corrected alignment information suitable for multi-frame super-resolution processing is generated.

  In other words, the image reduction device according to the present invention generates alignment information from the processing frame to the preceding and succeeding frames using the original image sequence, and reduces the reduced image sequence obtained by reducing the original image sequence at a predetermined image reduction rate. Generate. Furthermore, the image reduction apparatus according to the present invention uses the generated registration information and the reduced image sequence to enlarge the processed sample of the reduced image sequence at an enlargement rate corresponding to the image reduction rate. Generates an enlarged image by performing pixel interpolation using a point spread function for the position, calculates a difference value between the generated enlarged image pixel and the original image pixel, and spreads the value of the alignment information as a point spread Alignment information generated using the original image sequence is calculated by calculating a shift amount that minimizes the difference value while shifting within a range not exceeding the function width, and adding this shift amount to the value of the alignment information. The corrected alignment information is used as auxiliary information. The image enlargement apparatus according to the present invention acquires the corrected alignment information together with the reduced image sequence as auxiliary information, and performs pixel interpolation using the point spread function used in the image reduction device to enlarge the reduced image sequence. . As a result, it is possible to generate a multi-frame super-resolution image with higher image quality than in the prior art.

  In other words, the image reduction device of the present invention is an image reduction device that generates a reduced image sequence by reducing an original image sequence, and inputs an original image sequence consisting of a plurality of continuous original image frames, An alignment information generating unit that generates alignment information from the original image frame to at least one continuous original image frame, an image reducing unit that generates a reduced image sequence obtained by reducing the original image sequence at a predetermined image reduction rate, Using the generated alignment information and the reduced image sequence, when expanding the reduced image frame to be processed in the reduced image sequence at an enlargement rate corresponding to the image reduction rate, a point spread function is applied to the enlarged sample position. The enlarged image is generated by performing the pixel interpolation used, the difference value between the generated enlarged image pixel and the original image pixel is calculated, and the value of the alignment information is shifted within the processing range of the point spread function. Naga A positioning information correction unit that calculates a shift amount that minimizes the difference value, and adds the shift amount to a value of the positioning information to generate information obtained by correcting the positioning information as auxiliary information; It is characterized by providing.

  In the image reduction device of the present invention, the alignment information generation unit calculates the time-direction spectral power of the original image sequence to determine the number of frequency decomposition layers in the spatial direction, and the space corresponding to the number of frequency decomposition layers Information that is hierarchically aligned until the spatial resolution of the original image is achieved as the alignment information is generated.

  In the image reduction device of the present invention, the alignment information generation unit includes a first alignment processing unit that generates alignment information in units of blocks obtained by dividing the original image frame into blocks, and small areas in each block. A second alignment processing unit that generates alignment information in units of sub-blocks, and the alignment information detected in units of sub-blocks is greater than or equal to a predetermined threshold with respect to the amount of motion in units of blocks at the corresponding block positions, and / or When the coincidence evaluation function value is equal to or greater than a predetermined threshold, the accuracy is low if the accuracy is low, and the alignment information accuracy determination unit that performs accuracy determination that is effective if the accuracy is high otherwise, and the sub that is determined to be effective By correcting the alignment information in block units by adding the vector to the corresponding alignment information in block units, the alignment information is corrected. And correcting the positioning information generation unit that generates positioning information used in parts, characterized in that it comprises a.

  In the image reduction device of the present invention, the alignment information correction unit uses a motion amount adaptive point spread function in which the width of the point spread function changes according to the alignment information generated by the alignment information generation unit. Then, information obtained by correcting the alignment information is generated as auxiliary information.

  Furthermore, the image enlarging apparatus of the present invention uses the auxiliary information to align the reduced image sequence generated by the image reducing apparatus of the present invention at an enlargement ratio corresponding to the image reduction ratio. A pixel complementing unit that pixel-complements the enlarged sample point after the alignment, and an enlarged image generation unit that generates an enlarged image sequence corresponding to the original image sequence using the enlarged image after pixel complementation, It is characterized by providing.

  The program of the present invention inputs an original image sequence composed of a plurality of continuous original image frames to a computer that functions as an image reduction device that reduces the original image sequence to generate a reduced image sequence. Generating alignment information from the frame to at least one original image frame continuous; generating a reduced image sequence obtained by reducing the original image sequence at a predetermined image reduction ratio; and the generated alignment information and reduced image When enlarging a reduced image frame to be processed in a reduced image sequence at an enlargement rate corresponding to the image reduction rate using the sequence, enlargement is performed by performing pixel interpolation using a point spread function on the enlarged sample position An image is generated, a difference value between the generated enlarged image pixel and the original image pixel is calculated, and the value of the alignment information is shifted within the processing range of the point spread function. Calculating a shift amount that minimizes the difference value, and adding the shift amount to the value of the alignment information to generate information obtained by correcting the alignment information as auxiliary information. It is a program.

  The program of the present invention includes a step of aligning a reduced image sequence generated by the image reducing apparatus of the present invention with a magnification ratio corresponding to the image reduction ratio using the auxiliary information. , A program for performing pixel interpolation on the enlarged sample point subjected to the alignment, and generating an enlarged image sequence corresponding to the original image sequence using the enlarged image after pixel interpolation It is.

  According to the present invention, for the multi-frame super-resolution processing, it becomes possible to use post-correction alignment information that corrects not only the original image accuracy but also the restoration process error due to the point spread function. In addition, a high-resolution super-resolution image can be obtained.

  Further, by using a motion amount adaptive point spread function in which the width of the point spread function changes according to the alignment information of the original image accuracy, it is possible to generate more accurate corrected alignment information.

1 is a block diagram of an image reducing apparatus according to an embodiment of the present invention. It is a block diagram of the alignment information generation part in the image reduction device of one Example by this invention. It is a block diagram of the image reduction part in the image reduction device of one Example by this invention. It is a block diagram of the alignment information correction part in the image reduction device of one Example by this invention. 1 is a block diagram of an image enlargement apparatus according to an embodiment of the present invention. It is a block diagram of the alignment information generation part of another example in the image reduction device of one Example by this invention. It is a block diagram of the time direction high frequency domain power calculation part in the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. It is a block diagram of the spatial direction low frequency area | region power calculation part in the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. It is a block diagram of the alignment start resolution determination part in the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. It is a block diagram of the hierarchical alignment processing part in the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. It is an operation | movement flow which shows operation | movement of the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. It is a figure which shows the example of calculation of the power for every time direction frequency band of the frame image sequence in the alignment information generation part of the other example in the image reduction apparatus of one Example by this invention. (A), (b), (c), and (d) are examples in which a frame image sequence in another alignment information generation unit in the image reduction device according to the embodiment of the present invention is decomposed into Nmax floors in the time direction. FIG. (A), (b) is a figure which shows the example of the two-dimensional 2nd-order discrete wavelet in the alignment information generation part of another example in the image reduction apparatus of one Example by this invention. (A), (b), (c), (d) are the power ratio and alignment in the low frequency region in the spatial direction in the alignment information generating unit of another example in the image reduction device of one embodiment according to the present invention. It is a figure which illustrates the relationship of a start floor number. (A), (b), (c), (d) is a figure which shows the example of a relationship between the alignment start rank and block size in the alignment information generation part of another example in the image reduction apparatus of one Example by this invention. It is. It is a block diagram of the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. It is an operation | movement flowchart of the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. (A), (b) is a figure explaining the calculation example of the alignment information of the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. It is a figure explaining the accuracy determination example of the alignment information production | generation part of another example in the image reduction apparatus of one Example by this invention. FIG. 2 is a block diagram of a motion amount adaptive type point spread function generating device that generates a point spread function that can be used in the image reducing device according to the embodiment of the present invention. It is explanatory drawing of the process of the movement amount adaptive type point spread function generator which produces | generates the point spread function which can be used with the image reduction apparatus of one Example by this invention.

  Hereinafter, an image reduction apparatus and an image enlargement apparatus according to an embodiment of the present invention will be described in order with reference to the drawings.

(Image reduction device)
FIG. 1 is a block diagram of an image reducing apparatus according to an embodiment of the present invention. The image reduction device 10 according to the present exemplary embodiment includes a control unit 20 and a storage unit 30. The control unit 20 includes an alignment information generation unit 201, an image reduction unit 202, and an alignment information correction unit 203. The image reduction apparatus 10 of the present embodiment can be configured as a computer, and a program for causing the computer to realize each component according to the present invention is stored in a storage unit 30 provided inside or outside the computer. Is done. The control unit 20 provided in the computer can be realized by control of a central processing unit (CPU) or the like. That is, the CPU can appropriately read from the storage unit 30 a program in which the processing content for realizing the function of each component is described, and realize the function of each component on the computer. Here, the function of each component may be realized by a part of hardware. As will be apparent from the description below, the storage unit 30 stores “block division size information”, “image reduction rate information”, “mother wavelet information”, and “point spread function information”.

  The alignment information generation unit 201 inputs an original image sequence including a plurality of continuous original image frames, generates alignment information from the original image frame to be processed to at least one original image frame, The information is sent to the alignment information correction unit 203. Details of the alignment information generation unit 201 will be described later with reference to FIG.

  The image reduction unit 202 generates a reduced image sequence obtained by reducing the original image sequence at a predetermined image reduction rate, sends the reduced image sequence to the outside, and sends it to the alignment information correction unit 203. Details of the image reduction unit 202 will be described later with reference to FIG.

  The registration information correction unit 203 uses the generated registration information and the reduced image sequence to enlarge the reduced image frame to be processed in the reduced image sequence at an enlargement rate corresponding to the image reduction rate. Pixel enlargement using a point spread function is performed on the position to generate an enlarged image, a difference value between the generated enlarged image pixel and the original image pixel is calculated, and the value of the alignment information is set to the point spread While shifting within the processing range of the function, the shift amount that minimizes the difference value is calculated, and by adding the shift amount to the value of the alignment information, information obtained by correcting the alignment information is generated as auxiliary information. And send it to the outside. Details of the alignment information correction unit 203 will be described later with reference to FIG.

  FIG. 2 is a block diagram of an alignment information generation unit in the image reduction device according to the embodiment of the present invention. The alignment information generation unit 201 includes a block division unit 2011 and a block matching processing unit 2012.

  The block division unit 2011 reads the block division size information stored in the storage unit 30 using the original image at the processing frame position in the input original image sequence as a reference frame image, and is designated by the block division size information. Divide into blocks of Bx × By size.

  The block matching processing unit 2012 performs sub-pixel precision block matching on the base frame image obtained by dividing the block, using the original image at the previous and subsequent frame positions as a reference frame image, and performs alignment information for each block (that is, motion by block matching). Information corresponding to the information) is output. As a typical decimal pixel precision block matching, there is a parabolic fitting method. Further, as will be described later, alignment information can be generated by hierarchical alignment processing.

  FIG. 3 is a block diagram of the image reducing unit in the image reducing apparatus according to the embodiment of the present invention. The image reduction unit 202 includes a wavelet decomposition unit 2021, a spatial low frequency component extraction unit 2022, and a reduced image generation unit 2023.

  The wavelet decomposition unit 2021 is an example in which image reduction is performed using an n-th order spatial wavelet decomposition method (n is a natural number including 0). The wavelet decomposition unit 2021 reads out the mother wavelet information stored in advance in the storage unit 30 (that is, information on the filter coefficient indicating the decomposition coefficient of the mother wavelet) and information on the image reduction ratio, and stores the original frame position at the processing target position. Wavelet decomposition is performed at a resolution corresponding to the image reduction rate using the mother wavelet information using the image as a reference frame image, and the wavelet decomposed image is sent to the spatial low-frequency component extraction unit 2022.

  The spatial low-frequency component extraction unit 2022 extracts a low-frequency component of the wavelet-decomposed image (for example, an image of a low-frequency component in the horizontal and vertical directions when n = 1 and first-order spatial wavelet decomposition is performed). Then, it is sent to the reduced image generation unit 2023.

  The reduced image generation unit 2023 generates a reduced image corresponding to the extracted low frequency component. As a result, the original image sequence input to the image reduction unit 202 is converted into a reduced image sequence. Note that since the mother wavelet is variable and the mother wavelet is changed, the amount of aliasing distortion included in the reduced image sequence can be controlled. In this case, variable mother wavelet information is stored in the storage unit 30 as mother wavelet information. Store.

  FIG. 4 is a block diagram of an alignment information correction unit in the image reduction device of one embodiment according to the present invention. The alignment information correction unit 203 includes an all-pixel comparison processing unit 2031, an addition unit 2034, and an auxiliary information generation unit 2035, and the all-pixel comparison processing unit 2031 includes a pixel interpolation processing unit 2032 and a complementary error calculation unit. 2033.

  The pixel complementation processing unit 2032 uses the generated alignment information and the reduced image sequence to enlarge the processed frame of the reduced image sequence at an enlargement rate corresponding to the image reduction rate, and performs an enlargement on the sample position. An enlarged image is generated by performing pixel interpolation using a point spread function, and is sent to the complementary error calculation unit 2033 together with the original image corresponding to the processing target frame of the reduced image sequence.

  The complementary error calculation unit 2033 calculates a difference value between the generated pixel of the enlarged image and the pixel of the original image, and sets the value of the alignment information to the pixel complementary processing unit 2032 within a range not exceeding the point spread function width. The shift amount that minimizes the difference value is calculated while shifting, and is sent to the adder 2034. The difference value between the pixel of the generated enlarged image and the pixel of the original image is calculated for all the pixels of the generated enlarged image, whereby the minimum shift amount is calculated for each pixel.

  The adding unit 2034 adds the minimum shift amount calculated for each pixel to a value for each pixel of the alignment information (in the same block, the same value is given to each pixel). It is sent to the auxiliary information generation unit 2035 as corrected position adjustment information corrected in pixel units.

  The auxiliary information generation unit 2035 generates, as auxiliary information, including the generated corrected alignment information and the block division size information, the image reduction ratio information, and the point spread function information stored in the storage unit 30, and externally Send it out. As a result, it is possible to generate auxiliary information in which the alignment information generated using the original image sequence is corrected to the alignment information considering the point spread function in pixel interpolation. Note that the mother wavelet information can be included in the auxiliary information for the purpose of performing enlargement processing using wavelet reconstruction on the image enlargement apparatus side. However, an example in which the mother wavelet information is not used in the image enlarging apparatus 50 according to an embodiment described later will be described.

  Next, an image enlarging apparatus according to an embodiment of the present invention will be described.

[Image enlargement device]
FIG. 5 is a block diagram of an image enlargement apparatus according to an embodiment of the present invention. The image enlargement apparatus 50 according to the present embodiment aligns the reduced image sequence using the auxiliary information at an enlargement rate corresponding to the image reduction rate, and sets the enlarged sample points after the alignment as pixels. A device that complements and generates an enlarged image sequence corresponding to the original image sequence using the enlarged image after pixel complementation, and includes a control unit 60 and a storage unit 70. The control unit 60 includes an alignment processing unit 601, a pixel complement processing unit 602, and an enlarged image generation unit 603. The image enlargement apparatus 50 according to the present embodiment can be configured as a computer, and a program for causing the computer to realize each component according to the present invention is stored in a storage unit 70 provided inside or outside the computer. Is done. The control unit 70 provided in the computer can be realized by control of a central processing unit (CPU) or the like. That is, the CPU can appropriately read from the storage unit 70 a program in which the processing content for realizing the function of each component is described, and realize the function of each component on the computer.

  The registration processing unit 601 receives the reduced image sequence and the auxiliary information including the corrected registration information, the block division size information, the image reduction ratio information, and the point spread function information from the image reduction apparatus 10. For the reduced image at the position of the frame to be processed in the image sequence, the enlarged image sequence corresponding to the reduced image sequence (however, the sampling points that need to be complemented) at the enlargement rate corresponding to the image reduction rate using the input image reduction rate information Further, for each enlarged image in this enlarged image sequence, the pixel value in each block based on the block division size information is obtained from each reduced image sequence based on the corrected registration information. Information indicating pixel correspondence is generated and sent to the pixel complementing processing unit 602.

  The pixel complement processing unit 602 performs pixel complementation on each enlarged image in the generated enlarged image sequence using information indicating pixel correspondence from each reduced image sequence and point spread function information, and sequentially generates enlarged images. To the unit 603.

  The enlarged image generation unit 603 configures an enlarged image sequence made up of each enlarged image subjected to pixel interpolation and sends it out. Thereby, the enlarged image sequence corresponding to the original image sequence can be restored with high accuracy.

  Next, another example of the alignment information generation unit will be described with reference to FIGS.

(Generation of alignment information)
The case where the wavelet transform is used for frequency analysis in the time direction and the spatial direction as the alignment information generation unit 201b will be described. For the frequency analysis in the time direction and the space direction, other orthogonal transforms or FFT (Fast Fourier transform) can be used besides the case of using the wavelet transform.

[Device configuration]
FIG. 6 is a block diagram of another example of the alignment information generation unit in the image reduction apparatus according to the embodiment of the present invention. The alignment information generation unit 201b of this example calculates the time direction spectral power of the original image sequence to determine the number of frequency resolution layers in the spatial direction, and the spatial resolution of the original image with the spatial resolution according to the number of frequency resolution layers. It has the function to generate the information that has been hierarchically aligned until the position information, specifically, the time direction high frequency region power calculation unit 2011b, the spatial resolution rank determination unit 2012b, the spatial direction low frequency An area power calculation unit 2013b, an alignment start resolution determination unit 2014b, and a hierarchical alignment processing unit 2015b are provided.

The time-direction high-frequency domain power calculation unit 2011b includes a reference frame F (t _C ) for performing alignment processing and a reference frame F (t _R ) used for searching for alignment information at time t = t ₀ ... T _m frame image sequence _F (t ₀₎ of a plurality of frames _{in, ···, F (t C)} , ···, F (t R), ···, enter the F _{(t m),} the reference frame F ( For all the pixels at t _C ), after decomposing the plurality of frames into a frequency region of the maximum rank defined in advance in the time direction, the power for each frequency band in the time direction in all pixels is calculated, and the calculated time direction in all the pixels The ratio of the power in the high-frequency region in the time direction is calculated from the power for each frequency band and sent to the spatial resolution rank determining unit 2012b.

For example, as illustrated in FIG. 7, the time-direction high-frequency domain power calculation unit 2011b includes a frame image at time t = t ₀ ... T _m including a base frame F (t _C ) and a reference frame F (t _R ). column _{F (t 0), ···,} F (t C), ···, F (t R), ···, all the pixels of the F _{(t m)} enter the reference frame F _{(t C)} Time-direction one-dimensional Nmax-order discrete wavelet decomposition processing unit 2111b that performs Nwave-order (for example, fourth-order) discrete wavelet decomposition in advance in the time direction, and frequency in the time direction for all pixels of the reference frame F (t _C ) A time method for calculating the power for each band, calculating the ratio of the power in the high frequency region in the time direction from the calculated power for each frequency band in the time direction, and sending it to the spatial resolution rank determining unit 2012b It can be composed of a frequency band-dependent power calculator 2112 b.

  The spatial resolution rank determination unit 2012 has a larger dynamic region area as the proportion of power in the high frequency region in the time direction increases from the proportion of power in the high frequency region in the time direction calculated by the time direction high frequency region power calculation unit 2011b. It is determined that the amount of motion is large, the spatial frequency decomposition rank Ns (that is, the resolution in the spatial direction) is determined according to the power ratio of the high frequency region in the time direction, and information on the determined spatial frequency decomposition rank Ns is stored in the space. It sends out to the direction low frequency area power calculation part 2013b.

The spatial direction low frequency domain power calculation unit 2013b inputs information on the base frame F (t _C ) and the reference frame F (t _R ) and the spatial frequency decomposition rank Ns, and based on the spatial frequency decomposition rank Ns, The spatial Ns-order discrete wavelet decomposition is performed on the frame F (t _C ) and / or the reference frame F (t _R ), the power for each spatial frequency band in the frame is calculated, and the calculated power for each spatial frequency band Then, the power ratio of the low frequency region in the spatial direction in the frame is calculated, and the calculated power ratio of the low frequency region in the spatial direction and the spatial Ns-order discrete wavelet decomposition data are sent to the alignment start resolution determining unit 2014b. .

It should be noted that performing the spatial Ns-order discrete wavelet decomposition on both the base frame F (t _C ) and the reference frame F (t _R ) requires a fixed block size and a search range size as later processing. By hierarchically performing wavelet reconstruction when performing hierarchical alignment processing, it is advantageous in that hierarchical alignment processing with variable block size and search range size can be performed on the original image. In particular, in order to determine the resolution for performing the alignment processing hierarchically, the ratio of the power in the low frequency region in the spatial direction of the base frame F (t _C ) and the reference frame F (t _R ) It is preferable to select the larger one. In the following description, an example will be described in which spatial Ns-order discrete wavelet decomposition is performed on both the base frame F (t _C ) and the reference frame F (t _R ).

For example, as illustrated in FIG. 8, the spatial direction low frequency region power calculation unit 2013b receives the reference frame F (t _C ) and the reference frame F (t _R ), and the spatial frequency determined by the spatial resolution rank determination unit 2012b. A spatial direction two-dimensional Ns-order discrete wavelet decomposition processing unit that performs spatial Ns-order discrete wavelet decomposition on all the pixels of the base frame F (t _C ) and the reference frame F (t _R ) based on the decomposition rank Ns of and 2131B, the reference frame F _{(t C)} and the reference frame F _{(t R)} of calculating a power of each spatial frequency bands in each reference frame F _{(t C)} from the power of each calculated spatial frequency band and the reference frame F is calculated the ratio of the power of the low-frequency region of the spatial direction in each of the (t _R), the calculated reference frame F (t _C) and reference Can be composed of frame F (t _R) of each spatial direction each frequency band power calculation unit 2132b to be transmitted to the alignment start resolution determining unit 2014b of the larger information of the proportion of power in the low frequency region of the spatial direction in the .

  The alignment start resolution determination unit 2014b determines from the information on the power ratio of the low frequency region in the spatial direction calculated by the spatial direction low frequency region power calculation unit 2013b that the higher the power ratio of the low frequency region in the spatial direction (the spatial The resolution for starting the alignment process hierarchically (hereinafter referred to as “alignment start resolution”) is determined as the moving region area is large and the amount of motion is large as the power ratio of the high frequency region in the direction is small. The registration start resolution is determined according to the power ratio of the low-frequency region in the spatial direction, and information on the determined hierarchical registration start resolution is obtained. It is sent to the hierarchical alignment processing unit 2015b.

  For example, as illustrated in FIG. 9, the alignment start resolution determination unit 2014b determines the power of the low frequency region in the spatial direction from the ratio of the power of the low frequency region in the spatial direction calculated by the spatial direction low frequency region power calculation unit 2013b. Is larger (the smaller the power ratio of the high-frequency region in the spatial direction is), the moving area is larger and the amount of movement is larger, and the number of floors where alignment is started (hereinafter referred to as “alignment start floor”) The registration start floor number ns is determined according to the power ratio of the low frequency region in the spatial direction so that ns becomes a large value, and information on the determined alignment start floor number ns is used as the hierarchical alignment processing unit 2015b. It can be configured as an alignment start floor number determination unit 2141b to be sent to.

The hierarchical alignment processing unit 2015b obtains images of the reference frame F (t _C ) and the reference frame F (t _R ) in the low frequency region in the spatial direction according to the alignment start rank ns in order to obtain the reference frame F ( The spatial ns-order wavelet is reconfigured according to the alignment start rank ns for the spatially two-dimensional Ns-order discrete wavelet-decomposed data of t _C ) and the reference frame F (t _R ), and a predetermined block size is obtained. Then, the registration processing is executed with the size of the search range, the space ns-1 floor wavelet is reconstructed, and the registration processing is executed again with the predetermined block size and the size of the search range. The number of floors until the registration information is detected with the predetermined block size and the size of the search range in the upper layer (that is, the original image level) Decrement to repeat. In the operation of this hierarchical registration processing, the reference frame F (t _C ) and the reference frame F (t _R ) are input, and the block size is sequentially set with respect to the reference frame F (t _C ) based on the alignment start resolution. This process is similar to performing the alignment process while reducing the size of the search range. However, according to the hierarchical alignment processing by sequentially repeating through the spatial ns-order wavelet decomposition and reconstruction, it is not necessary to prepare the block size and the search range size that should be sequentially changed according to the hierarchy, and are fixed. In addition, since the alignment process according to the alignment start floor number ns according to the image scene is performed, high accuracy can be expected.

For example, as illustrated in FIG. 10, the hierarchical alignment processing unit 2015b includes images of the reference frame F (t _C ) and the reference frame F (t _R ) in the low frequency region in the spatial direction according to the alignment start rank ns. In order to obtain the registration information, the spatial ns floor wavelet is reconfigured according to the registration start rank ns, and alignment information generation is performed by block matching with decimal pixel precision with a predetermined block size and search range size. The spatial Ns floor calculated by the spatial direction low frequency region power calculation unit 2013b in order to decrement the floor until the original image level corresponding to the highest floor is repeated. Empty so that the discrete wavelet decomposition data has a higher rank image than the alignment start rank ns. It can be configured from a spatial first-floor wavelet reconstruction unit 2152b that performs wavelet reconstruction on the first floor in the inter-direction and sends it to the alignment information generation unit 2151b. Therefore, the registration information generation unit 2151b performs hierarchical registration processing using the reconstructed images of the reference frame F (t _C ) and the reference frame F (t _R ) obtained from the spatial first-order wavelet reconstruction unit 2152b. The final alignment information can be determined and output repeatedly.

  Hereinafter, the operation of the alignment information generation unit of this example will be described in more detail.

[Device operation]
FIG. 11 is an operation flow showing the operation of another example of the alignment information generating unit in the image reducing apparatus according to the embodiment of the present invention. In step S1, the alignment information generation unit 201b includes the frame image sequence F (t ₀ ) at time t = t ₀ ... T _m including the base frame F (t _C ) and the reference frame F (t _R ). ,..., F (t _C ),..., F (t _R ),..., F (t _m ) are input, and a storage unit (not shown) included in the alignment information generation unit 201b. Are stored in a readable manner.

In step S2, the temporal high-frequency range power calculation unit 2011 b, frame image sequence _{F (t 0), ···,} F (t C), ···, F (t R), ···, F ( t _m ) is input, and all the pixels in the reference frame F (t _C ) are decomposed into frequency regions of the maximum rank defined in advance in the time direction, and then power for each frequency band in the time direction in all pixels is calculated. The ratio of the power in the high-frequency region in the time direction is calculated from the power for each frequency band in the time direction in all the pixels.

For example, as shown in FIG. 12, frame image sequence _{F (t 0), ···,} F (t C), ···, F (t R), ···, one at F _{(t m)} After the pixel R (k, l) is decomposed into frequency regions of the maximum rank (Nmax) defined in advance in the time direction, the power for each time direction frequency band in all the pixels can be calculated. For example, as shown in FIG. 13, if a frame image sequence F (t) of 16 frames is decomposed into Nmax floors in the time direction, when Nmax = 1, it is divided into a low frequency region L ¹ and a high frequency region H ^1. (See FIG. 13A), when Nmax = 2, it can be divided into the low frequency region L ² and the high frequency regions H ¹ and H ² (see FIG. 13B), and when Nmax = 3, it is low. It can be divided into the frequency region L ³ and the high frequency regions H ¹ , H ² , H ³ (see FIG. 13C), and when Nmax = 4, the low frequency region L ⁴ and the high frequency regions H ¹ , H ² , H ³ and H ⁴ (see FIG. 13D).

In step S3, when the power ratio of the high frequency region in the time direction is larger from the ratio of power in the high frequency region in the time direction calculated by the spatial resolution rank determining unit 2012b, the moving region area is larger and the amount of motion is larger. The spatial frequency decomposition rank of the quasi-frame F (t _C ) and the reference frame F (t _R ) is determined according to the power ratio in the high-frequency region in the time direction so that the spatial frequency decomposition rank Ns becomes a large value. Ns is determined.

  For example, as shown in Table 1, a table defined in advance between the power ratio in the high frequency region in the time direction and the resolution rank Ns of the spatial frequency is held in advance.

In step S4, the base frame F (t _C ) and the reference frame F (t _R ) are input by the spatial direction low frequency region power calculation unit 2013b, and the spatial frequency decomposition rank Ns determined by the spatial resolution rank determination unit 2012b. based on the reference frame F _{(t C)} and the reference frame F _{(t R)} performs spatial Ns floor discrete wavelet decomposition on the reference frame F _{(t C),} and the spatial frequency in the reference frame F _{(t R)} The power for each band is calculated, and the reference frame F (t _C ) and the reference frame F (t _R ) are calculated from the power for each calculated spatial frequency band in order to determine the alignment start resolution (or alignment start floor ns). The power ratio for each spatial frequency band in (1) is selected. Only the power ratio for each spatial frequency band in the reference frame F (t _C ) or the reference frame F (t _R ) may be calculated.

For example, as shown in FIG. 14A, the power of each frequency domain is calculated by performing two-dimensional second-order discrete wavelet decomposition in the spatial direction on all the pixels of the reference frame F (t _C ). The power ratio in the low frequency region in the spatial direction can be calculated from the power for each spatial frequency band. Further, as shown in FIG. 14 (b), the reference frame F _{(t C)} a low-frequency region (e.g., LL ²⁾ spatial direction only extracted by the reference frame F low-frequency region only of the _{(t C)} Images can be reconstructed.

  In step S5, the alignment start resolution determination unit 2014b determines that the power ratio of the low frequency region in the spatial direction is larger (space direction) from the spatial Ns-order discrete wavelet decomposition data and the power ratio of the low frequency region in the spatial direction. The direction of the spatial direction is such that the smaller the power ratio of the high-frequency region is, the larger the moving region area is, and the larger the amount of motion, the lower the alignment start resolution (the value at which the alignment start floor ns is large). Hierarchical alignment start resolution (or alignment start rank ns) is determined according to the power ratio of the low frequency region. However, ns ≦ Ns.

  For example, as shown in Table 2, a table defined between the power ratio of the low frequency region in the spatial direction and the alignment start resolution (or alignment start floor ns) is held in advance. This means that the resolution of the original image is reduced as the alignment start floor number increases, and the block size and the size of the search range for alignment information increase relative to the original image. It means that. For example, if the spatial resolution is 1/16, 1/8, 1/4, 1/2, the (block size, the size of the search range of the alignment information) is (16 × 16, horizontal / vertical 16 pixels). ), (8 × 8, horizontal / vertical 8 pixels), (4 × 4, horizontal / vertical 4 pixels), (2 × 2, horizontal / vertical 2 pixels), and the like. Here, for example, the spatial resolution of 1/16 means a reduction in resolution in which 16 pixels are sampled as one pixel at the horizontal sampling frequency Hs and the vertical sampling frequency Vs in the original image.

That is, as shown in FIG. 15, the alignment start rank ns can be associated with the ratio of the power in the low frequency region in the spatial direction. For example, when Ns = 4, the respective powers of the low frequency region (LL ⁴ ) and the high frequency region (other than LL ⁴ ) are calculated, and the ratio of the low frequency region (LL ⁴ ) in the whole is 99.5% or more. If so, it is possible to reconstruct an image of only the low-frequency region of the four layers with the alignment start rank ns = 4 (see FIG. 15D). Further, if the ratio of the low frequency region (LL ⁴ ) in the whole is 98.0% or more and less than 99.5%, the alignment start rank ns = 3 and the three layers of low frequency regions (in this case, LL ³ ). Only an image can be reconstructed (see FIG. 15C). Similarly, when the ratio of the low frequency region (LL ⁴ ) in the whole is 95.0% or more and less than 98.0%, the alignment start rank ns = 2 is set to two layers of low frequency regions (in this case, LL ² ) Only can be reconstructed (see FIG. 15B), and if the ratio of the low frequency region (LL ⁴ ) in the whole is less than 95.0%, the alignment start rank ns = 1 An image of only one layer of the low-frequency region (in this case, LL ¹ ) can be reconstructed (see FIG. 15A).

In step S6, the hierarchical alignment processing unit 2015b applies the low-frequency images of the reference frame F (t _C ) and the reference frame F (t _R ) based on the alignment start resolution (alignment start rank ns). The low-frequency image of the reference frame F (t _C ) is divided into a predetermined block size, and for each divided block, registration information by block matching with decimal pixel accuracy is performed with the size of a search range of predetermined alignment information. The generation process is performed.

In step S7, the hierarchical alignment processing unit 2015b performs alignment while reducing the block size and the size of the search range of the alignment information with respect to the base frame F (t _C ) and the reference frame F (t _R ). In order to obtain the effect of repeating the processing, the ranks higher than the alignment start rank ns with respect to the spatial ns-order discrete wavelet decomposition data of the calculated base frame F (t _C ) and reference frame F (t _R ) Perform wavelet reconstruction on the first floor in the spatial direction so that an image is obtained.

The hierarchical alignment processing unit 2015b uses the reconstructed images of the reference frame F (t _C ) and the reference frame F (t _R ) obtained from the spatial first-order wavelet reconstruction unit 2152b, and uses the highest layer (that is, The rank is sequentially decremented until it reaches the original image level alignment information) and the alignment process is repeated (step S8), and final alignment information can be determined and output (step S9).

For example, as shown in FIGS. 16A to 16D, the block size increases as the alignment start rank ns increases, and the alignment in the reference frame F (t _R ) increases as the block size increases. The size of the information search range also increases.

That is, the ns-order low-frequency image of the reference frame F (t _C ) is divided into a certain block B ^ns (the superscript indicates the class) of horizontal and vertical block sizes (Bx ^ns , By ^ns ) and referred to ± ^{Sx ns} frame F _{(t R),} probed with a range of ± ^{Sy ns} (e.g., ± 2 blocks), calculates the alignment information ^{v ns} of each block. Next, the ns-1 floor low frequency image of the base frame F (t _C ) is divided by the block size (Bx ^ns , By ^ns ), and the same position on the ns floor low frequency image of the reference frame F (t _R ) 2 × v ^ns shifted by location respectively ± Sx ns as the horizontal and vertical number of pixels centered position ^from probed with a range of ± Sy ^ns, the 2 × v ^ns to the obtained positioning information by vector addition , Ns-1 floor low-frequency image registration information v ^ns-1 is calculated. In this way, high accuracy can be achieved by repeating the alignment process up to the highest rank (that is, the first floor).

  Note that the alignment processing is preferably performed only with the highest rank (that is, the first floor) using the block matching method of decimal pixel positions by quadratic function approximation, and is given by the following equation: .

   When the pixel position at the search position is x, SSD (x) represents the SSD value (sum of squares of error) at the pixel position. More specifically, SSD (0) is the SSD value at the center position, SSD (−1) represents the SSD value at the position of −Sx (Sy) pixel from the center position, and SSD (1) represents the SSD value at the position of + Sx (Sy) pixel from the center position. From the above equations, the pixel position (decimal pixel position) with decimal pixel accuracy in the horizontal or vertical direction can be calculated.

  In this way, alignment information can be generated based on the frequency power in the spatiotemporal direction.

  Next, another example of the alignment information generating unit in the image reducing apparatus according to the embodiment of the present invention will be described.

(Accuracy judgment of alignment information)
FIG. 17 is a block diagram of another example of the alignment information generating unit in the image reducing apparatus according to the embodiment of the present invention. The alignment information generating unit 201c in this example generates alignment information in units of blocks obtained by dividing the original image frame, generates alignment information in units of sub-blocks of small areas in each block, and in units of sub-blocks. If the detected alignment information is greater than or equal to a predetermined threshold with respect to the amount of motion of the corresponding block position in units of blocks and / or the matching score evaluation function value is greater than or equal to a predetermined threshold, the accuracy is low and invalid. Otherwise, the accuracy is determined to be valid if the accuracy is high, and the alignment information for each sub-block determined to be valid is corrected by adding the vector to the alignment information for the corresponding block to correct the alignment information. It has a function of generating alignment information used by the correction unit 203, specifically, a block unit decimal pixel alignment processing unit 20 Comprising 1c and the pixel unit decimal alignment processing unit 2012c, and the alignment information accuracy determining unit 2013c, and a correction positioning information generation unit 2014C. Note that the alignment information generation unit 201c of this example includes a storage unit (not shown) that stores data necessary for each component to process. _Let F (t _C ) be a reference frame for performing alignment processing, and F (t _R ) be a reference frame for searching for alignment information.

The block unit decimal pixel alignment processing unit 2011c calculates alignment information for each block of the original image sequence. More specifically, as will be described later, the block unit decimal pixel alignment processing unit 2011c receives the base frame F (t _C ) and the reference frame F (t _R ) and inputs the base frame F (t _C ). Is divided into blocks B _{i, j} of 16 × 16 pixels (where i, j are horizontal and vertical block numbers), and the reference frame F (t _R ) is searched to obtain block-unit alignment information v _{Bi, j} is obtained and held in the storage unit. It is preferable that the alignment processing for each block is performed with alignment processing with decimal pixel accuracy.

The pixel unit decimal pixel alignment processing unit 2012c calculates alignment information in units of sub-blocks (for example, in units of one pixel) in a small area in the block B _{i, j} . More specifically, as will be described later, the pixel unit decimal pixel alignment processing unit 2012c performs alignment information u _{i, j} () on the pixel position (k, l) in the block B _{i, j} . k, l) is obtained and held in the storage unit. It is preferable to perform alignment processing with sub-pixel accuracy for the sub-block unit (for example, one pixel unit) alignment information of the small area.

  The alignment information accuracy determination unit 2013c reads the alignment information detected in units of blocks and sub-blocks stored in the storage unit, and the alignment information detected in units of sub-blocks indicates the movement of the corresponding block position in units of blocks. When the quantity is greater than or equal to a predetermined threshold and / or the coincidence evaluation function value is greater than or equal to the predetermined threshold, the accuracy is determined to be invalid if the accuracy is low, and the accuracy is determined to be valid if the accuracy is high otherwise. The alignment information accuracy determination unit 2013c corrects the alignment information that indicates whether the alignment information obtained by accuracy determination is valid for the sub-block unit alignment information and the alignment information detected in block units and sub-block units. The information is sent to the information generation unit 2014c. More specifically, the alignment information accuracy determination unit 2013c determines that the alignment information detected in units of sub-blocks is equal to or greater than a half value of the amount of motion in units of blocks, and / or the coincidence evaluation function value is constant. If it is greater than or equal to the threshold value, the accuracy is invalid and invalid. Otherwise, the accuracy is high and valid.

  The corrected alignment information generating unit 2014c corrects the alignment information in units of sub-blocks determined to be valid by adding the vector to the corresponding alignment information in units of blocks.

  As a result, the alignment information obtained in units of blocks is re-detected in units of sub-blocks (for example, in units of pixels) of a smaller area, and whether the alignment information in units of sub-blocks (for example of units of pixels) is valid or invalid Therefore, highly accurate alignment information can be obtained. In addition, since alignment processing with decimal pixel accuracy is performed at least in sub-block units (for example, pixel units) in a small area, alignment information with high accuracy can be obtained.

  Hereinafter, with reference to FIG. 18, the operation of another example of the alignment information generation unit in the image reduction device according to the embodiment of the present invention will be described in detail.

In step S11, the base frame F (t _C ) and the reference frame F (t _R ) constituting the original image sequence are input and stored in the storage unit as appropriate.

In step S12, the block unit decimal pixel alignment processing unit 2011c first generates alignment information with decimal pixel accuracy in block units (for example, block units of 16 × 16 pixels). For example, as shown in FIG. 19A, the reference frame F (t _C ) is divided into blocks B _{i, j} (i and j are horizontal and vertical block numbers), and as shown in FIG. 19B. Next, the reference frame F (t _R ) is searched to obtain block unit alignment information v _{Bi, j} .

  In step S13, the pixel unit decimal pixel alignment processing unit 2012c performs an alignment process with decimal pixel accuracy in sub-block units (for example, pixel units) in the small area. Hereinafter, an example of performing the alignment process with decimal pixel accuracy in pixel units will be described.

The pixel position (k, l) in the block B _{i, j} is used, and the alignment information u _{i, j} (k, l) of each pixel position is used as described above, and the alignment information at the decimal pixel precision position in pixel units is used. Is generated. The pixel-by-pixel alignment process uses the block matching method that uses a degree-of-match evaluation function using quadratic function approximation (parabolic fitting), and performs block-unit alignment information v _Bi, on the reference frame F (t _R ) _. Search is performed in a range of ± Sx ′ (Sy ′) centered on a position shifted by _j . The size of ± Sx ′ or Sy ′ that defines the size of the search range can be ± 1 pixel.

In step S14, the alignment information u _{i, j} (k, l) obtained in pixel units by the alignment information accuracy determination unit 2013c is equal to or greater than a predetermined threshold with respect to the block-unit motion amount. If the degree evaluation function value is greater than or equal to a predetermined threshold value, the accuracy is low and invalid, and otherwise, the accuracy is high and valid (steps S15 and S16).

For example, the accuracy is low when the obtained alignment information u _{i, j} (k, l) in pixel units is equal to or greater than the half value of the motion amount in blocks, that is, ± (0.5,0.5) or more. Is invalid, and the others are valid with high accuracy (see FIG. 20).

As yet another example, when the obtained pixel unit alignment information u _{i, j} (k, l) is greater than or equal to a certain threshold value (for example, the value of SSD (0)), the accuracy is high. Invalid if low, otherwise valid with high accuracy.

As yet another example, when the obtained pixel unit alignment information u _{i, j} (k, l) is equal to or greater than the half value of the block-unit motion amount, that is, ± (0.5,0.5), and When the SSD value (sum of squared errors) is equal to or greater than a certain threshold (for example, the value of SSD (0)), the accuracy is low and invalid, and otherwise, the accuracy is high and valid.

When it is determined to be valid, the correction alignment information generation unit 2014c performs alignment information V (i, j, k, l) in sub-block units (for example, pixel units) in the small area in the reference frame F (t _C ). ) _Is corrected as V (i, j, k, l) = v _{Bi, j} + u _{i, j} (k, l).

Note that the correction alignment information generation unit 2014c replaces the alignment information of the target pixel unit decimal accuracy position that is invalid, with the surrounding effective value (for example, the most adjacent effective value), or By substituting with a plurality of effective values in the surrounding area or replacing with alignment information in block units (ie, V (i, j, k, l) = v _{Bi, j} ) It is also possible to determine alignment information V (i, j, k, l) in block units (for example, pixel units).

  As described above, with the alignment information generation unit 201c of the present embodiment, the alignment information obtained in units of blocks is re-detected with decimal precision in sub-block units (for example, pixel units) of smaller areas, Since validity / invalidity determination is performed on the alignment information in units of sub-blocks, alignment information with high accuracy and high accuracy can be obtained.

  Next, the motion amount adaptive point spread function will be described.

(Motion-adaptive point spread function)
FIG. 21 is a block diagram of a motion amount adaptive point spread function generating apparatus that generates a point spread function that can be used in the image reducing apparatus according to an embodiment of the present invention. Usually, a point spreading function PSF (Point Spreading Function) is used as a motion blur function generated at the time of image reduction, and super-resolution is performed using the inverse function at the time of image enlargement. Therefore, the motion amount adaptive type point spread function generating device 300 generates registration information using the original image sequence, and the motion amount adaptive type point spread function in which the point spread function changes adaptively according to the registration information. Is an example of generating. The motion amount adaptive type point spread function can be used by storing in advance in the storage unit 30 in the image reduction device 10 described above.

  The motion amount adaptive type point spread function generating apparatus 300 uses the motion amount adaptive type point spread function in which the width of the point spread function changes according to the alignment information generated by the alignment information generating unit 201, and the alignment information. Can be incorporated into the alignment information correction unit 203. Specifically, the motion amount adaptive point spread function generation device 300 includes a registration information generation unit 301, a pixel association processing unit 302, and a reconstruction processing unit 303.

  The alignment information generation unit 301 generates alignment information using the original image sequence, as in the case described with reference to FIG.

  The pixel association processing unit 302 performs pixel association between a plurality of frames on the reduced image sequence generated in the same manner as described with reference to FIG. 1 from the alignment information generated using the original image sequence. Association information that reflects the pixels of the resolution source frame (frame of the reduced image sequence) on the super-resolution frame (frame for enlarging the image) is generated and sent to the reconstruction processing unit 303 (see FIG. 21).

  The reconstruction processing unit 303 uses a posteriori probability when reflecting a reduced image pixel on a frame for enlarging an image (when a plurality of low-resolution super-resolution source frames are provided on the super-resolution frame). The motion amount adaptive type point spread function necessary for reconstructing the high-resolution image is determined by estimating the high-resolution image that maximizes the posterior probability). For this estimation, a MAP (Maximum A Poster) method or other methods can be used. The motion amount adaptive type point spread function is a motion blur function obtained by extending the PSF based on the alignment information generated using the original image sequence. As will be described in detail later, the reconstruction processing unit 303 uses a two-dimensional Gaussian function approximation that simulates a point spread function PSF as a motion amount adaptive point spread function, and performs horizontal and vertical motions of the two-dimensional Gaussian function approximation. The amount of blur can be controlled.

A conventional MAP method (hereinafter, referred to as “conventional MAP method”) is a method in which x is a frame image vector restored with a high resolution image, y is a super resolution original frame image vector with a low resolution, and N is the number of input frames. This is a method for estimating a high-resolution frame image that maximizes the posterior probability p (x | y ₁ , y ₂ ,..., Y _N ).

In the first term in equation (1), D is a matrix representing sub-sampling, B _k is a point spread function PSF matrix, and W _k is also applicable to alignment information (especially a motion amount representing a scene or camera motion). ) Matrix.

Therefore, || y _k −D · B _k · W _k · x || ² of the first term of Equation (1) is the k-th low-resolution super-resolution source frame image and the high resolution as the observation signal. This is a square error between the sub-sampled image given the motion blur at the time of resolution conversion by PSF and the alignment information from the kth super-resolution source frame to the frame image to be restored.

Λ || C · x || ^{2 in} the second term of equation (1) is “image reduction rate information” in which λ is a variable amount indicating the degree of contribution of prior information, and C is given as prior information of the input image. It becomes the matrix of.

  However, it should be noted that in the conventional MAP method, “point spread function PSF”, “alignment information”, and “image reduction ratio information” are not provided as auxiliary information. Therefore, C in the conventional MAP method assumes a MRF (Markov Random Field) in the vicinity of 4, for example, and uses a Laplacian kernel in the vicinity of 4 so that it cannot sufficiently cope with the scene and the movement of the camera. .

The motion amount adaptive type point spread function generating apparatus 300 of this example has a B of the point spread function PSF of Expression (1) under the situation where “positioning information” and “image reduction rate information” are not given as auxiliary information. _k is a point spread function by two-dimensional Gaussian function approximation, and the point spread function is expanded by two-dimensional Gaussian function approximation according to the alignment information (that is, information corresponding to the direction and magnitude of the inter-frame motion vector). Do. This expanded PSF function is defined as a motion amount adaptive point spread function kernel EB _k .

Formula (2) shows a MAP reconstruction processing formula using the motion amount adaptive point spread function kernel EB _k .

  More specifically, the “point spread function PSK by two-dimensional Gaussian function approximation” and the “motion amount adaptive point spread function (point spread function PSK by extended two-dimensional Gaussian function approximation)” will be described in detail.

[Point spread function by two-dimensional Gaussian function approximation]
When approximating PSK with a two-dimensional Gaussian function, the half-value width is defined as w and the amplitude of a normalized Gaussian function with an area of 1 is defined as α, as shown in Equation (3).

From Equation (3), the two-dimensional Gaussian function is as shown in Equation (4). Here, the horizontal reference position is x ₀ , and the vertical reference position is y ₀ .

[Motion-adaptive point spread function (point spread function based on an extended 2D Gaussian function)]
The motion amount adaptive point spread function generator 300 expands the point spread function of Equation (4). Horizontal movement amount m _x alignment information _m, the movement amount in the vertical direction and m _y, the half-width of the horizontal and vertical directions in a Gaussian function of Equation (4), as the formula (5) Be controlled.

  Expression (5) controls the spread direction and spread amount of the point spread function according to the direction and size of the alignment information.

In the reconstruction processing of the motion amount adaptive type point spread function generation device 300, the kernel EB _k of the equation (2) is used as a two-dimensional Gaussian function of the equation (5), and the half-value width of the equation (5) according to the image enlargement magnification. It is possible to determine an optimum high-resolution image by determining w and performing the repetitive calculation of Expression (2) while changing λ of Expression (2).

  As described above, when the motion amount adaptive point spread function generated by the motion amount adaptive point spread function generation device 300 is applied to the image reduction device 10 shown in FIG. 1, sharp alignment information for the motion amount is generated. be able to.

  According to the present invention, for the multi-frame super-resolution processing, it becomes possible to use post-correction alignment information that corrects not only the original image accuracy but also the restoration process error due to the point spread function. In addition, a high-resolution super-resolution image can be obtained, which is useful for obtaining an enlarged image from a reduced image.

DESCRIPTION OF SYMBOLS 10 Image reduction apparatus 20 Control part 30 Storage part 50 Image enlargement apparatus 60 Control part 70 Storage part 201, 201b, 201c Registration information generation part 202 Image reduction part 203 Registration information correction part 300 Motion amount adaptive type point spread function generation apparatus 301 Registration Information Generation Unit 302 Pixel Association Processing Unit 303 Reconstruction Processing Unit 601 Registration Processing Unit 602 Pixel Complement Processing Unit 603 Enlarged Image Generation Unit 2011 Block Division Unit 2012 Block Matching Processing Unit 2021 Wavelet Decomposition Unit 2022 Spatial Low Frequency Component Extraction unit 2023 Reduced image generation unit 2031 All pixel comparison processing unit 2032 Pixel interpolation processing unit 2033 Complementary error calculation unit 2034 Addition unit 2035 Auxiliary information generation unit 2011b Time-direction high-frequency domain power calculation unit 2012b Spatial resolution rank determination Unit 2013b spatial direction low frequency region power calculation unit 2014b alignment start resolution determination unit 2015b hierarchical alignment processing unit 2011c block unit decimal pixel alignment processing unit 2012c pixel unit decimal alignment processing unit 2013c alignment information accuracy determination unit 2014c correction Positioning information generator

Claims (7)

An image reduction device that generates a reduced image sequence by reducing an original image sequence,
An alignment information generation unit that inputs an original image sequence including a plurality of continuous original image frames and generates alignment information from the original image frame to be processed to at least one original image frame;
An image reduction unit for generating a reduced image sequence obtained by reducing the original image sequence at a predetermined image reduction rate;
Using the generated alignment information and the reduced image sequence, when expanding the reduced image frame to be processed in the reduced image sequence at an enlargement rate corresponding to the image reduction rate, a point spread function is applied to the enlarged sample position. The enlarged image is generated by performing the pixel interpolation used, the difference value between the generated enlarged image pixel and the original image pixel is calculated, and the value of the alignment information is shifted within the processing range of the point spread function. While calculating the shift amount that minimizes the difference value, and adding the shift amount to the value of the alignment information, the alignment information correction unit that generates information that has corrected the alignment information as auxiliary information,
An image reduction apparatus comprising:   The alignment information generating unit calculates the time-direction spectral power of the original image sequence to determine the number of frequency resolution layers in the spatial direction, until the spatial resolution of the original image becomes the spatial resolution corresponding to the number of frequency resolution layers The image reduction apparatus according to claim 1, wherein information that is hierarchically aligned is generated as the alignment information. The alignment information generation unit
A first alignment processing unit for generating alignment information in units of blocks obtained by dividing the original image frame;
A second alignment processing unit that generates alignment information in units of sub-blocks of small areas in each block;
When the alignment information detected in units of sub-blocks is greater than or equal to a predetermined threshold with respect to the amount of motion in units of the corresponding block position and / or the matching score evaluation function value is greater than or equal to a predetermined threshold, the accuracy is A positioning information accuracy determination unit that performs accuracy determination that invalidates as low and is effective as high accuracy otherwise;
Corrected alignment information generation for generating alignment information to be used by the alignment information correction unit by correcting the alignment information of sub-block units determined to be valid by adding the vector to the corresponding alignment information of blocks. And
The image reduction device according to claim 1, further comprising: The alignment information correction unit
Using the motion amount adaptive type point spread function in which the width of the point spread function changes according to the alignment information generated by the alignment information generation unit, and generating the corrected information as the auxiliary information. The image reduction device according to claim 1, wherein the image reduction device is a feature. An alignment process for aligning the reduced image sequence generated by the image reduction device according to any one of claims 1 to 4 at an enlargement rate corresponding to the image reduction rate using the auxiliary information. And
A pixel complementing unit that pixel-complements the enlarged sample point after the alignment;
An enlarged image generating unit that generates an enlarged image sequence corresponding to the original image sequence using an enlarged image after pixel interpolation;
An image enlarging apparatus comprising: In a computer functioning as an image reduction device that reduces the original image sequence to generate a reduced image sequence,
Inputting an original image sequence composed of a plurality of continuous original image frames and generating alignment information from the original image frame to be processed to at least one original image frame;
Generating a reduced image sequence obtained by reducing the original image sequence at a predetermined image reduction rate;
Using the generated alignment information and the reduced image sequence, when expanding the reduced image frame to be processed in the reduced image sequence at an enlargement rate corresponding to the image reduction rate, a point spread function is applied to the enlarged sample position. The enlarged image is generated by performing the pixel interpolation used, the difference value between the generated enlarged image pixel and the original image pixel is calculated, and the value of the alignment information is shifted within the processing range of the point spread function. While calculating the shift amount that minimizes the difference value and adding the shift amount to the value of the alignment information to generate information that corrects the alignment information as auxiliary information;
A program for running On the computer,
Aligning the reduced image sequence generated by the image reduction device according to any one of claims 1 to 4 at an enlargement rate corresponding to the image reduction rate using the auxiliary information;
Pixel complementation of the enlarged sample point subjected to the alignment; and
Generating an enlarged image sequence corresponding to the original image sequence using an enlarged image after pixel interpolation;
A program for running

Images