JP5894422B2 - Spatio-temporal low correlation image super-resolution device, image space super-resolution device, and program thereof - Google Patents

Description

  The present invention includes a spatiotemporal low correlation image super-resolution device that generates a super-resolution image of a spatio-temporal low correlation image, and the spatio-temporal low correlation image super-resolution device, and generates a super-resolution image of an input image. The present invention relates to an image space super-resolution apparatus and their programs.

  In recent years, high-definition imaging devices and display devices have advanced, and high-definition video such as a digital cinema called a so-called 4K system or super high-definition video such as Super Hi-Vision (SHV) called an 8K system has been conventionally used. The high resolution (HDTV) video is 4 to 16 times as high as the video. A high-definition image has an enormous amount of information. Therefore, a high-resolution image called super-resolution, in which a transmission device transmits a low-resolution image and a reception device generates a high-resolution image from the low-resolution image. Resolution technology is being studied.

  For example, a point (temporary corresponding point) is set in advance on the conversion target frame, a pixel (corresponding pixel) on another frame corresponding to each temporary corresponding point is obtained, and the position of the temporary corresponding point is determined based on the corresponding pixel. Correct to match the corresponding pixel, or obtain an accurate subpixel position on the image of another frame, interpolate and generate a pixel value corresponding to that position, and other available when calculating the corresponding pixel Set an appropriate range for all of the frames, make each pixel a candidate for the corresponding pixel, and leave only those candidates that are sufficiently likely to be supported (relatively reliable). By eliminating other (relatively less reliable) candidates, only the relatively reliable candidates can be left out of all other frames, reducing the possibility of noise Possible resolution conversion Location is known (e.g., see Patent Document 1).

  Also, the sampling phase difference is estimated using the image data on the reference input image frame and the corresponding image data on other input image frames, the sampling phase difference is converted, and the converted sampling phase difference is output. Then, using the information of the converted sampling phase difference, the image data of each input image frame is motion compensated and the number of pixels is increased, and the image data of each image frame with the increased number of pixels is phase-shifted by a predetermined amount, 2. Description of the Related Art A resolution conversion apparatus that removes aliasing components and outputs a super-resolution image by multiplying each image data before and after phase shift by a coefficient determined using sampling phase difference information is known. (For example, refer to Patent Document 2).

JP 2010-34696 A JP 2009-94593 A

  In the conventional technique, super-resolution processing is performed for all components of the original image without considering image correlation. However, this technique has a problem in that sufficient super-resolution image quality cannot be obtained for pixel components having a low spatio-temporal correlation such as sharp edges and isolated point noise.

  The present invention has been made in view of the above-described problems, and a spatio-temporal low-correlation image super-resolution capable of generating a high-resolution super-resolution image even for an image including a spatio-temporal low-correlation component. An object is to provide an image device, an image space super-resolution device, and a program thereof.

  In order to solve the above-described problem, a spatiotemporal low correlation image super-resolution apparatus according to the present invention performs super-resolution processing on a spatiotemporal low correlation image composed of pixel components having a low spatiotemporal correlation. A device that performs multi-resolution analysis of a spatiotemporal low correlation image, generates a high frequency component image of the spatiotemporal low correlation image, and detects a pixel position where the pixel value of the high frequency component image is negative A phase adjustment unit that adjusts the phase of the spatio-temporal low-correlation image by changing a tap position element to which a multi-resolution analysis filter is applied to a pixel at the same pixel position as the detected pixel position; A frequency component reconstruction unit that performs super-resolution processing using the low-correlation image and the spatio-temporal low-correlation image adjusted in phase to generate a super-resolution image of the spatio-temporal low-correlation image. It is characterized by.

  Further, in the spatio-temporal low correlation image super-resolution apparatus according to the present invention, the spatial frequency decomposition unit generates a high-frequency component image by Haar wavelet decomposition of the spatio-temporal low correlation image in the spatial direction without decimation, The phase adjustment unit detects a pixel position where the pixel value of the high-frequency component image is negative, and applies a Haar wavelet filter to the pixel at the same pixel position as the detected pixel position for the spatiotemporal low correlation image Generating a spatio-temporal low correlation image in which the phase is adjusted by exchanging elements of tap positions to be performed, and the frequency component reconstructing unit generates the spatiotemporal low correlation image and the spatiotemporal low correlation image in which the phase is adjusted. And a Haar wavelet reconstruction is performed to generate a super-resolution image of the spatio-temporal low correlation image.

  In order to solve the above problems, an image space super-resolution apparatus according to the present invention generates an image space super-resolution image of an input image using the spatio-temporal low correlation image super-resolution apparatus described above. A spatiotemporal correlation component separating unit that separates an input image into a spatiotemporal low correlation image composed of pixel components having a low spatiotemporal correlation and a spatiotemporal high correlation image composed of pixel components having a high spatiotemporal correlation; The spatio-temporal low-correlation image super-resolution device that generates a spatio-temporal low-correlation super-resolution image by super-resolution processing of the spatio-temporal low-correlation image separated by the spatio-temporal correlation component separation unit, and the spatio-temporal correlation component A spatio-temporal highly correlated image super-resolution unit that generates a spatio-temporal highly correlated super-resolution image by super-resolution processing of the spatio-temporal highly correlated image separated by the separating unit; Super-resolution images are generated by adding and synthesizing spatio-temporal highly correlated super-resolution images Characterized in that it comprises an image synthesizing unit.

Furthermore, in the image space super-resolution apparatus according to the present invention, the spatio-temporal correlation component separation unit performs multi-resolution analysis of the input image in the time direction, and is a temporal low frequency component that is an image having a low frequency component in the time direction. A temporal frequency decomposition unit that generates a temporal high-frequency component image, which is an image and an image having a high-frequency component in the time direction, and multi-resolution analysis in the spatial direction for the temporal high-frequency component image, and a low-frequency component in the spatio-temporal direction A spatial frequency decomposition unit that generates a primary spatio-temporal low-frequency component image that is an image having a high-frequency component in a spatio-temporal direction, and the primary spatio-temporal high-frequency component image Is generated in the spatiotemporal direction to generate the spatiotemporal low correlation image, and the temporal low frequency component image and the primary spatiotemporal low frequency component image are reconstructed in the spatiotemporal direction to generate the spatiotemporal high frequency image. The frequency component reconstruction unit for generating a related image,
It is characterized by providing.

Furthermore, in the image space super-resolution apparatus according to the present invention, the spatiotemporal correlation component separation unit calculates an isolated point component level indicating a size of an isolated point component included in the first-order spatiotemporal high frequency component image, An image composed of pixels below the isolated point component level in the primary space-time high-frequency component image is generated as a secondary space- time high-frequency component image, and from the pixels exceeding the isolated point component level in the primary space- time high-frequency component image. A spatio-temporal correlation component detection unit that generates a secondary spatio-temporal low-frequency component image, and the frequency component reconstructing unit reconstructs the secondary spatio-temporal high-frequency component image in a spatio-temporal direction, and A spatiotemporal low correlation image is generated, and the spatiotemporal high correlation image is reconstructed in the spatiotemporal direction by reconstructing the temporal low frequency component image, the primary spatiotemporal low frequency component image, and the secondary spatiotemporal high frequency component image. Generate a And wherein the Rukoto.

  In order to solve the above problems, a spatio-temporal low correlation image super-resolution program according to the present invention causes a computer to function as the above-described spatio-temporal low correlation image super-resolution apparatus.

  In order to solve the above problems, an image space super-resolution program according to the present invention causes a computer to function as the above-described image space super-resolution apparatus.

  According to the present invention, a high-resolution super-resolution image can be generated even for an image including a spatiotemporal low correlation component.

It is a block diagram which shows the structural example of the spatio-temporal low correlation image super-resolution apparatus based on this invention. It is a figure explaining operation | movement of the phase adjustment part in the spatio-temporal low correlation image super-resolution apparatus based on this invention. It is a block diagram which shows the structural example of the image space super-resolution apparatus which concerns on this invention. It is a block diagram which shows the 1st structural example of the spatiotemporal correlation component separation part in the image space super-resolution apparatus which concerns on this invention. It is a figure explaining the process example of the spatiotemporal correlation component separation part in the image space super-resolution apparatus which concerns on this invention. It is a block diagram which shows the 2nd structural example of the spatiotemporal correlation component separation part in the image space super-resolution apparatus which concerns on this invention. It is a figure explaining the isolated point extraction process of the spatiotemporal correlation component separation part in the image space super-resolution apparatus which concerns on this invention. It is a block diagram which shows the structural example of the spatiotemporal highly correlated image super-resolution part in the image space super-resolution apparatus which concerns on this invention. It is a figure explaining operation | movement of the spatiotemporal highly correlated image super-resolution part in the image space super-resolution apparatus which concerns on this invention. It is a block diagram which shows the structural example of the image synthetic | combination part in the image space super-resolution apparatus which concerns on this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  A spatio-temporal low correlation image super-resolution apparatus according to the present invention will be described below. FIG. 1 is a block diagram showing a configuration example of a spatiotemporal low correlation image super-resolution apparatus according to the present invention. As shown in FIG. 1, the spatiotemporal low correlation image super-resolution apparatus 10 includes a spatial frequency decomposition unit 11, a phase adjustment unit 12, and a frequency component reconstruction unit 13. The spatio-temporal low correlation image super-resolution processing unit 10 performs super-resolution processing on the spatio-temporal low correlation image and generates a spatio-temporal low correlation super-resolution image. Since the spatiotemporal low correlation image is an image with high autocorrelation, super-resolution is performed by a method that is not easily affected by surrounding pixels.

  The spatial frequency decomposition unit 11 performs multi-resolution analysis (for example, wavelet decomposition) on the spatio-temporal low correlation image that is an input image in the spatial direction, and generates a low frequency component image of the spatiotemporal low correlation image and a spatiotemporal low correlation image. A high frequency component image is generated. Then, the spatiotemporal low correlation image and the high frequency component image of the spatiotemporal low correlation image are output to the phase adjustment unit 12.

  The phase adjustment unit 12 detects a pixel position where the pixel value of the high-frequency component image generated by the spatial frequency decomposition unit 11 is negative, and the spatio-temporal low correlation image is applied to a pixel at the same pixel position as the detected pixel position. On the other hand, the element of the tap position (for example, the tap position to which the wavelet filter is applied) to which the multi-resolution analysis filter is applied is exchanged to generate a spatiotemporal low correlation image with the phase adjusted. Then, the spatiotemporal low correlation image and the phase adjusted spatiotemporal low correlation image are output to the frequency component reconstruction unit 13.

  The frequency component reconstructing unit 13 performs super-resolution processing (reconstruction) using the spatio-temporal low correlation image and the spatio-temporal low correlation image phase-adjusted by the phase adjustment unit 12 to perform spatio-temporal low correlation super-resolution. Generate an image.

  In the following, an example of generating a super-resolution image of a spatio-temporal low correlation image by Haar wavelet decomposition on the first floor and Haar wavelet reconstruction on the first floor will be shown. The spatial frequency decomposition unit 11 performs first-order Haar wavelet decomposition on a spatio-temporal low correlation image (hereinafter referred to as “input image F” in the present embodiment) without decimation in the spatial direction, and generates low-frequency component images LL, A horizontal high-frequency component image LH, a vertical high-frequency component image HL, and an oblique high-frequency component image HH are generated.

  The phase adjustment unit 12 detects a pixel position where the pixel value is negative for the image LH of the horizontal high-frequency component generated by the spatial frequency decomposition unit 11. Then, for the input image F, a first phase adjustment image F1 is generated by replacing the element at the tap position to which the Haar wavelet filter is applied to the pixel position at the same position as the detected pixel position. Since the number of taps of the Haar wavelet filter is 2, adjacent pixels in the horizontal direction are replaced.

  Further, the phase adjustment unit 12 detects a pixel position where the pixel value is negative in the image HL of the high frequency component in the vertical direction generated by the spatial frequency decomposition unit 11. And about the input image F, the 2nd phase adjustment image F2 which replaced the element of the tap position which applies a Haar wavelet filter with respect to the pixel position of the same position as this detected pixel position is produced | generated. Since the number of taps of the Haar wavelet filter is 2, adjacent pixels in the vertical direction are replaced.

  Further, the phase adjustment unit 12 detects a pixel position at which the pixel value is negative for the image HH of the oblique high-frequency component generated by the spatial frequency decomposition unit 11. Then, for the input image F, a third phase adjustment image F3 is generated by replacing the tap position element to which the Haar wavelet filter is applied to the pixel position at the same position as the detected pixel position. Since the number of taps of the Haar wavelet filter is 2, pixels adjacent in the diagonal direction are replaced.

  Then, the phase adjustment unit 12 outputs the generated first phase adjustment image F1, second phase adjustment image F2, and third phase adjustment image F3 together with the input image F to the frequency component reconstruction unit 13. . Note that the phase adjustment unit 12 may not perform phase adjustment on the oblique high-frequency component (that is, do not generate the second phase adjustment image F2).

  FIG. 2 is a diagram for explaining the operation of the phase adjustment unit 12. FIG. 2A shows an input image F, a horizontal high-frequency component image LH, a vertical high-frequency component image HL, and an oblique high-frequency component image HH, which are images input to the phase adjustment unit 12. . The minus in the figure means that the value of the pixel is negative. FIG. 2B shows an input image F, a first phase adjustment image F1, a second phase adjustment image F2, and a third phase adjustment image F3, which are images output from the phase adjustment unit 12. . The arrows in the figure mean that the pixel values are exchanged.

  Assuming that an image reduced by Haar wavelet decomposition is reconstructed by Haar wavelet decomposition, the pixel position of the high-frequency component image generated by Haar wavelet decomposition is negative for the high-frequency component after Haar wavelet reconstruction. It is known that the phase of the pixel is reversed from that of the adjacent pixel. Therefore, the phase adjustment unit 12 adjusts the phase as described above before the Haar wavelet reconstruction. That is, for the input image F, an image in which pixel values adjacent in the horizontal direction are replaced at a pixel position where the pixel value of the image LH of the horizontal high-frequency component is negative is defined as the first phase adjustment image F1. Similarly, for the input image F, an image in which pixel values adjacent to each other in the vertical direction are replaced at a pixel position where the pixel value of the image HL of the vertical high-frequency component is negative is referred to as a second phase adjustment image F2. In addition, regarding the input image F, an image in which pixel values adjacent in the diagonal direction are replaced at a pixel position where the pixel value of the image HH of the diagonal high frequency component is negative is referred to as a third phase adjustment image F3.

  The frequency component reconstruction unit 13 uses the input image F as a low frequency component, the first phase adjustment image F1 as a horizontal high frequency component, the second phase adjustment image F2 as a vertical high frequency component, and a third phase adjustment. The first-floor Haar wavelet is reconstructed from the image F3 as an oblique high-frequency component, and a spatio-temporal low-correlation super-resolution image is generated. When the phase adjustment unit 12 does not perform phase adjustment for the oblique high-frequency component, the frequency component reconstruction unit 13 sets the input image F as the low-frequency component and the oblique high-frequency component, and the first phase adjustment image. An image having F1 as a horizontal high-frequency component and the second phase adjustment image F2 as a vertical high-frequency component is reconstructed on the first floor to generate a spatiotemporal low-correlation super-resolution image.

  As described above, the spatiotemporal low correlation image super-resolution apparatus 10 performs multi-resolution analysis of the spatiotemporal low correlation image by the spatial frequency decomposition unit 11 to generate a high frequency component image of the spatiotemporal low correlation image, and performs phase adjustment. The tap position where the pixel position where the pixel value of the high frequency component image is negative is detected by the unit 12 and the multiresolution analysis filter is applied to the pixel at the same pixel position as the detected pixel position in the spatiotemporal low correlation image The phase of the spatiotemporal low correlation image is obtained by performing super-resolution processing using the spatiotemporal low correlation image and the spatiotemporal low correlation image after the phase adjustment by the frequency component reconstruction unit 13. A super-resolution image is generated. The spatio-temporal low correlation image super-resolution apparatus 10 adjusts the phase of the high-frequency component image by the phase adjustment unit 12, so that the quality of the super-resolution image of the spatio-temporal low correlation image can be improved.

  In particular, when Haar wavelet decomposition is used for multiresolution analysis, a pixel position where the pixel value of the high-frequency component image is negative is detected, and a pixel at the same pixel position as the detected pixel position is detected for the spatiotemporal low correlation image. In contrast, phase adjustment is performed by replacing the tap position elements to which the Haar wavelet filter is applied, and Haar wavelet reconstruction is performed using the spatiotemporal low-correlation image after phase adjustment to generate an optimal super-resolution image. can do.

  Note that a computer can be suitably used to function as the above-described spatio-temporal low correlation image super-resolution device 10, and such a computer realizes each function of the spatio-temporal low correlation image super-resolution device 10. It can be realized by storing a program describing the processing contents in a storage unit of the computer, and reading and executing the program by a CPU (central processing unit) of the computer.

  Next, the image space super-resolution apparatus according to the present invention will be described below. FIG. 3 is a block diagram showing a configuration example of the image space super-resolution apparatus according to the present invention. As shown in FIG. 3, the image space super-resolution device 1 includes a spatio-temporal low correlation image super-resolution device 10, a spatio-temporal correlation component separation unit 20, a spatio-temporal high correlation image super-resolution unit 30, and The image composition unit 40 is provided.

  The image space super-resolution device 1 generates a super-resolution image of the input image in consideration of the correlation degree of the input image. For this purpose, the input image is separated into an image composed of pixel components having a low spatiotemporal correlation (spatiotemporal low correlation image) and an image composed of pixel components having a high spatiotemporal correlation (spatiotemporal high correlation image). After performing the super-resolution processing in, it is synthesized.

  The spatiotemporal correlation component separation unit 20 separates the input image into a spatiotemporal low correlation image composed of pixel components having a low spatiotemporal correlation and a spatiotemporal high correlation image composed of pixel components having a high spatiotemporal correlation. The image is output to the spatio-temporal low correlation image super-resolution device 10, and the spatio-temporal high correlation image is output to the spatio-temporal high correlation image super-resolution unit 30. Details will be described later.

  The spatio-temporal low correlation image super-resolution apparatus 10 generates a spatio-temporal low correlation super-resolution image by performing super-resolution processing on the spatio-temporal low correlation image generated by the spatio-temporal correlation component separation unit 20 as described above. And output to the image composition unit 40.

  The spatiotemporal high correlation image super-resolution unit 30 generates a spatio-temporal highly correlated super-resolution image by performing super-resolution processing on the spatio-temporal high correlation image generated by the spatio-temporal correlation component separation unit 20. Output to 40. Details will be described later.

  The image synthesis unit 40 includes a spatio-temporal low correlation super-resolution image generated by the spatio-temporal low correlation image super-resolution device 10 and a spatio-temporal high correlation super-resolution image generated by the spatio-temporal high correlation image super-resolution unit 30. The image images are combined to generate a super-resolution image considering the spatiotemporal correlation. Details will be described later.

[Spatio-temporal correlation component separation unit]
Next, details of the spatio-temporal correlation component separation unit 20 will be described with reference to FIG. FIG. 4 is a block diagram illustrating a first configuration example of the spatiotemporal correlation component separation unit 20. As shown in FIG. 4, the spatiotemporal correlation component separation unit 20 includes a temporal frequency decomposition unit 21, a spatial frequency decomposition unit 22, and a frequency component reconstruction unit 23.

  The time-frequency decomposition unit 21 performs multi-resolution analysis (for example, wavelet decomposition) on the input image in the time direction, and has a time low-frequency component image that is an image having a low-frequency component in the time direction and a high-frequency component in the time direction. A temporal high-frequency component image that is an image is generated, the temporal high-frequency component image is output to the spatial frequency decomposition unit 22, and the temporal low-frequency component image is output to the frequency component reconstruction unit 23.

  The spatial frequency decomposition unit 22 performs multi-resolution analysis (for example, wavelet decomposition) in the spatial direction on the temporal high frequency component image generated by the temporal frequency decomposition unit 21 and is an image having a low frequency component in the spatiotemporal direction. A spatial low-frequency component image and a spatio-temporal high-frequency component image which is an image having a high-frequency component in the spatio-temporal direction are generated, and both are output to the frequency component reconstruction unit 23.

  FIG. 5 is a diagram for explaining processing examples of the time-frequency decomposition unit 21 and the spatial frequency decomposition unit 22. FIG. 5A shows a state in which the time-frequency decomposition unit 21 generates an image of a time-low frequency component and an image of a time-high-frequency component (shaded portion) by first-order wavelet decomposition of the input image in the time direction. Yes.

  In FIG. 5B, the spatial frequency decomposition unit 22 performs the second-order wavelet decomposition with the decimation in the spatial direction on the temporal high-frequency component image generated by the temporal frequency decomposition unit 21, and the spatio-temporal low-frequency component image; It shows how a spatio-temporal high-frequency component image (shaded area) is generated. When the n-th wavelet decomposition is performed in the spatial direction, 3n high-frequency components are generated.

The frequency component reconstructing unit 23 reconstructs the spatio-temporal high-frequency component image generated by the spatial frequency resolving unit 22 in the spatio-temporal direction (for example, wavelet reconstruction), and converts the image composed only of the high-frequency component into a spatio-temporal low correlation image. And output to the spatiotemporal low correlation image super-resolution apparatus 10.
The frequency component reconstruction unit 23 reconstructs the temporal low-frequency component image generated by the temporal frequency decomposition unit 21 and the spatio-temporal low-frequency component image generated by the spatial frequency decomposition unit 22 in the spatio-temporal direction (for example, , Wavelet reconstruction), and an image composed of only low frequency components is generated as a spatiotemporal high correlation image and output to the spatiotemporal high correlation image super-resolution section 30.

  FIG. 6 is a block diagram illustrating a second configuration example of the spatiotemporal correlation component separation unit 20. As shown in FIG. 6, the spatiotemporal correlation component separation unit 20 includes a temporal frequency decomposition unit 21, a spatial frequency decomposition unit 22, a spatiotemporal correlation component detection unit 24, and a frequency component reconstruction unit 23.

  The processes of the time frequency decomposition unit 21 and the spatial frequency decomposition unit 22 are the same as those in the first configuration example. The spatio-temporal correlation component detection unit 24 performs isolated wavelet decomposition in a region of the same size for each frequency band in the same manner, and performs an image of each frequency band generated by performing wavelet decomposition without decimation. May be the same size.

  The spatio-temporal high-frequency component image generated by the spatial frequency decomposition unit 22 is further separated into a high-frequency component and a low-frequency component by the spatio-temporal correlation component detection unit 24. Therefore, the spatiotemporal low frequency component image and the spatiotemporal high frequency component image generated by the spatial frequency decomposition unit 22 are referred to as a primary spatiotemporal low frequency component image and a primary spatiotemporal high frequency component image, and the spatiotemporal correlation component detection unit. The spatio-temporal low-frequency component image and the spatio-temporal high-frequency component image generated by 24 are referred to as a secondary spatio-temporal low-frequency component image and a secondary spatio-temporal high-frequency component image.

  The spatiotemporal correlation component detection unit 24 calculates an isolated point component level indicating the size of the isolated point component included in the primary spatio-temporal high-frequency component image generated by the spatial frequency decomposition unit 22 and calculates the primary spatio-temporal high-frequency component. An image composed of pixels below the isolated point component level is generated as a secondary spatio-temporal high-frequency component image, and an image composed of pixels exceeding the isolated point component level among the primary spatio-temporal high-frequency component images Generated as a frequency component image. Specifically, the spatiotemporal correlation component detection unit 24 includes an isolated point position extraction unit 241, an isolated point component level calculation unit 242, and a spatiotemporal correlation component analysis unit 243.

The isolated point position extraction unit 241 sets the pixel value of a pixel that exceeds a predetermined threshold Th ₁ to ₁ for the band component image of the primary spatio-temporal high-frequency component generated by the spatial frequency decomposition unit 22, and is equal to or less than the threshold Th _1. A binarized image B in which the pixel value of each pixel is 0 is generated. Threshold Th ₁ is, for example, the median or mean value of all element values of non-zero in each band components.

The isolated point position extraction unit 241 then calculates, for a pixel having a pixel value of 1 in the binarized image B, a total value of pixel values in a predetermined determination area centered on the pixel, and a predetermined threshold Th ₂ . Compare If the total value of the pixel value of the determination area of the binarized image B exceeds the threshold value Th _2, the total of the pixel is determined not to be an isolated point, the pixel value of the determination area of the binarized image B If the value is the threshold value Th ₂ or less determines the pixel as the isolated point. Then, the position information (isolated point pixel position information) of the pixel determined to be an isolated point is output to the isolated point component level calculation unit 242.

FIG. 7 is a diagram for explaining an operation example of isolated point determination of the isolated point position extraction unit 241 and shows an example of the binarized image B. In the example shown in FIG. 7, the determination area is 3 × 3 pixels. When the threshold value Th _2, and 1, only when pixel values of 8 pixels around the pixel having a pixel value of 1 of the binary image B is 0, the pixel values of the binarized image B pixel is 1 Considered as an isolated point. Therefore, when the threshold Th ₂ = 1, the pixel P1 in the figure is determined to be an isolated point, and the pixels P2 and P3 are not determined to be isolated points.

  The isolated point component level calculation unit 242 is based on the isolated point pixel position information input from the isolated point position extraction unit 241 for the image of each band component of the primary spatio-temporal high-frequency component generated by the spatial frequency decomposition unit 22. The pixel value of the pixel position determined as an isolated point is acquired, and a value based on the acquired pixel value (for example, the average value or median value of non-zero elements) is calculated as the isolated point component level of each band component, The result is output to the spatiotemporal correlation component analysis unit 243.

  The spatio-temporal correlation component analysis unit 243 selects an image composed of pixels having an isolated point component level or less calculated by the isolated point component level calculation unit 242 from the primary spatiotemporal high frequency component image generated by the spatial frequency decomposition unit 22. A secondary spatio-temporal high-frequency component image is generated as a secondary spatio-temporal high-frequency component image and an image composed of pixels exceeding the isolated point component level calculated by the isolated point component level calculation unit 242 among the primary spatio-temporal high-frequency component images. Generate as

  The frequency component reconstruction unit 23 reconstructs the secondary spatio-temporal high-frequency component image generated by the spatio-temporal correlation component analysis unit 243 in the spatio-temporal direction (for example, wavelet reconstruction), and generates an image composed only of the high-frequency components. A spatially low correlation image is generated and output to the spatiotemporal low correlation image super-resolution apparatus 10. In addition, the frequency component reconstruction unit 23 includes a temporal low-frequency component image generated by the temporal frequency decomposition unit 21, a primary spatio-temporal low-frequency component image generated by the spatial frequency decomposition unit 22, and a spatio-temporal correlation component analysis unit. The secondary spatio-temporal low-frequency component image generated by H.243 is reconstructed in the spatio-temporal direction (for example, wavelet reconstruction), and an image consisting of only the low-frequency component is generated as a spatio-temporal highly correlated image. The image is output to the image super-resolution unit 30.

[Spatio-temporal highly correlated image super-resolution part]
Next, details of the spatio-temporal highly correlated image super resolving unit 30 will be described with reference to FIGS. FIG. 8 is a block diagram illustrating a configuration example of the spatiotemporal highly correlated image super resolving unit 30. As shown in FIG. 8, the spatiotemporal highly correlated image super resolving unit 30 includes a spatial frequency resolving unit 31, a spatial high frequency component expanding unit 32, a spatial high frequency component smoothing unit 33, and a frequency component reconstructing unit 34. Is provided. FIG. 9 is a diagram for explaining the operation of the spatiotemporal highly correlated image super resolving unit 30.

  The spatial frequency decomposition unit 31 first-order wavelet-decomposes the spatio-temporal highly correlated image, which is an input image, in the spatial direction, the spatial low frequency component image LL, the horizontal high frequency component image LH, the vertical high frequency component image HL, And the image HH of a diagonal high frequency component is produced | generated (step S101). The high-frequency component image, that is, the horizontal high-frequency component image LH, the vertical high-frequency component image HL, and the diagonal high-frequency component image HH is output to the spatial high-frequency component enlargement unit 32.

  The spatial high-frequency component enlarging unit 32 enlarges (upsamples) each of the horizontal high-frequency component image LH, the vertical high-frequency component image HL, and the diagonal high-frequency component image HH generated by the spatial frequency decomposing unit 31. (Steps S102, 103, 104). The enlarged image of the horizontal high-frequency component image LH (horizontal high-frequency component enlarged image Ex.LH), the enlarged image of the vertical high-frequency component image HL (vertical high-frequency component enlarged image Ex.HL), and the oblique high-frequency component An enlarged image of the component image HH (oblique high-frequency component enlarged image Ex.HH) is output to the spatial high-frequency component smoothing unit 33.

  Specifically, as shown in FIG. 8, the spatial high-frequency component enlarging unit 32 enlarges the vertical high-frequency component image HL as a spatial low-frequency component to expand the vertical high-frequency component image HL. First-order wavelet reconstruction is performed in the spatial direction for an image with zero components, and a vertical high-frequency component enlarged image Ex. HL is generated. Similarly, when the image LH of the horizontal high-frequency component is enlarged, the first-order wavelet reconstruction is performed in the spatial direction with respect to the image in which the horizontal high-frequency component image LH is the spatial low-frequency component and the spatial high-frequency component is zero. The horizontal high-frequency component enlarged image Ex. LH is generated. When enlarging the image HH of the oblique high-frequency component, the first-order wavelet reconstruction is performed in the spatial direction with respect to the image with the spatial high-frequency component set to zero with the image HH of the oblique high-frequency component as the spatial low-frequency component. Oblique high-frequency component enlarged image Ex. HH is generated.

  The spatial high-frequency component smoothing unit 33 generates a horizontal direction high-frequency component enlarged image Ex. LH, vertical high-frequency component enlarged image Ex. HL and oblique high-frequency component enlarged image Ex. Each HH is smoothed by a smoothing filter (steps S105, 106, 107). Then, the smoothed horizontal high-frequency component enlarged image Ex. LH ′, vertical high-frequency component enlarged image Ex. HL ′ and the oblique high-frequency component enlarged image Ex. HH ′ is output to the frequency component reconstruction unit 34. For example, a Gaussian filter or a bilateral filter is used as the smoothing filter.

  The frequency component reconstructing unit 34 includes a horizontal direction high-frequency component enlarged image Ex. LH ′, vertical high-frequency component enlarged image Ex. HL ′ and the oblique high-frequency component enlarged image Ex. Reconstruction processing is performed using HH 'and the spatiotemporal highly correlated image that is the input image to generate a super-resolution image (step S108).

  Specifically, as illustrated in FIG. 9, the frequency component reconstructing unit 34 sets the spatiotemporal high correlation image, which is an input image, as a spatial low frequency component, and uses the horizontal direction high frequency component enlarged image Ex. LH ′, vertical high-frequency component enlarged image Ex. HL ′ and the oblique high-frequency component enlarged image Ex. A first-order wavelet reconstruction is performed in the spatial direction for an image in which HH ′ is a horizontal high-frequency component, a vertical high-frequency component, and a diagonal high-frequency component, respectively, and a super-resolution image is generated.

[Image composition part]
Next, details of the image composition unit 40 will be described. When the spatiotemporal correlation component separation unit 20 does not detect the isolated point component level as shown in FIG. 4, the image synthesis unit 40 generates the spatiotemporal space generated by the spatiotemporal low correlation image super-resolution device 10. The low-correlation super-resolution image and the spatio-temporal high-correlation super-resolution image generated by the spatio-temporal high-correlation image super-resolution unit 30 are added and synthesized to generate a final super-resolution image.

  On the other hand, when the spatiotemporal correlation component separation unit 20 detects the isolated point component level as shown in FIG. 6, the spatiotemporal low correlation super-resolution image is normalized using the isolated point component level. A configuration example of the image composition unit 40 in this case is shown in FIG. As shown in FIG. 10, the image composition unit 40 includes a level adjustment unit 41 and a composition unit 42.

  In the level adjustment unit 41, the RMS (Root Mean Square) value of the spatio-temporal low correlation super-resolution image generated by the spatio-temporal low correlation image super-resolution device 10 is detected by the isolated point component level calculation unit 242. Normalization is performed so as to be equal to the point component level value, and the spatiotemporal low correlation super-resolution image after normalization is output to the synthesis unit 42.

  The synthesizing unit 42 includes a spatio-temporal low correlation super-resolution image generated by the level adjustment unit 41 and a spatio-temporal high correlation super-resolution image generated by the spatio-temporal high correlation image super-resolution unit 30. Are combined to generate a final super-resolution image.

  As described above, the image-space super-resolution device 1 separates the input image into a spatio-temporal low-correlation image and a spatio-temporal high-correlation image by the spatio-temporal correlation component separation unit 20, and the spatio-temporal low-correlation image super-resolution device 10. A spatio-temporal low correlation image is generated by super-resolution processing to generate a spatio-temporal low correlation super-resolution image, and the spatio-temporal high correlation image super-resolution unit 30 performs super-resolution processing on the spatio-temporal high correlation image. A super-resolution image is generated, and the image synthesis unit 40 adds and synthesizes the spatio-temporal low correlation super-resolution image and the spatio-temporal high correlation super-resolution image to generate a super-resolution image. As a result, a pixel component having a low spatiotemporal correlation to a pixel component having a high spatiotemporal correlation can be super-resolution with high image quality, and the image quality of the super-resolution image can be improved.

  In the first configuration example, the spatio-temporal correlation component separation unit 20 reconstructs the primary spatio-temporal high-frequency component image extracted by the temporal frequency decomposition unit 21 and the spatial frequency decomposition unit 22 by the frequency component reconstruction unit 23. An image obtained by reconstructing an image composed of components other than the primary spatio-temporal high-frequency component image by the frequency component reconstruction unit 23 is a spatio-temporal highly correlated image. Thereby, the input image can be separated into a spatiotemporal low correlation image and a spatiotemporal high correlation image by a simple method.

  In the second configuration example, the spatiotemporal correlation component separation unit 20 further converts the primary spatiotemporal high frequency component image generated by the temporal frequency decomposition unit 21 and the spatial frequency decomposition unit 22 into a spatiotemporal correlation component detection unit 24. Compared with the isolated point component level, the image composed of pixels below the isolated point component level is used as a secondary spatio-temporal high-frequency component image, and the secondary spatio-temporal high-frequency component image is reconstructed by the frequency component reconstructing unit 23. A spatio-temporal low-correlation image is used, and an image obtained by reconstructing an image composed of components other than the secondary spatio-temporal high-frequency component image by the frequency component reconstruction unit 23 is a spatio-temporal high-correlation image. Thereby, an image with higher autocorrelation can be made a spatiotemporal low correlation image. In addition, since the image composition unit 40 can perform the spatiotemporal low correlation super-resolution image level adjustment using the isolated point component level, a higher-resolution super-resolution image can be generated.

  It should be noted that a computer can be suitably used to function as the above-described image space super-resolution device 1, and such a computer is a program describing processing contents for realizing each function of the image space super-resolution device 1. Is stored in the storage unit of the computer, and this program is read and executed by the CPU of the computer.

  Although the above-described embodiments have been described as representative examples, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes may be made without departing from the scope of the claims. Is possible.

  The present invention is useful for any application for generating a super-resolution image, for example, generation of a high-definition video such as a digital cinema or an ultra-high-definition video such as a super high-definition video.

DESCRIPTION OF SYMBOLS 1 Image space super-resolution apparatus 10 Spatio-temporal low correlation image super-resolution apparatus 11 Spatial frequency decomposition part 12 Phase adjustment part 13 Frequency component reconstruction part 20 Spatio-temporal correlation component separation part 21 Temporal frequency decomposition part 22 Spatial frequency decomposition part 23 Frequency component reconstruction unit 24 Spatio-temporal correlation component detection unit 30 Spatio-temporal highly correlated image super-resolution unit 31 Spatial frequency decomposition unit 32 Spatial high-frequency component enlargement unit 33 Spatial high-frequency component smoothing unit 34 Frequency component reconstruction unit 40 Image composition unit 41 level adjustment unit 42 synthesis unit 241 isolated point position extraction unit 242 isolated point component level calculation unit 243 spatio-temporal correlation component analysis unit

Claims (7)

A spatio-temporal low-correlation image super-resolution apparatus for super-resolution processing of a spatio-temporal low-correlation image composed of pixel components having a low spatio-temporal correlation,
A spatial frequency decomposition unit that performs multi-resolution analysis of a spatiotemporal low correlation image and generates a high frequency component image of the spatiotemporal low correlation image;
Tap position element that detects a pixel position where the pixel value of the high-frequency component image is negative and applies a multi-resolution analysis filter to a pixel at the same pixel position as the detected pixel position in the spatiotemporal low correlation image A phase adjustment unit that adjusts the phase by exchanging
A frequency component reconstruction unit that performs super-resolution processing using the spatio-temporal low correlation image and the phase-adjusted spatio-temporal low correlation image, and generates a super-resolution image of the spatio-temporal low correlation image; A spatio-temporal low-correlation image super-resolution apparatus comprising: The spatial frequency decomposition unit generates a high-frequency component image by performing Haar wavelet decomposition on a spatio-temporal low correlation image without decimation in the spatial direction,
The phase adjustment unit detects a pixel position where the pixel value of the high-frequency component image is negative, and applies a Haar wavelet filter to the pixel at the same pixel position as the detected pixel position for the spatiotemporal low correlation image To generate a spatio-temporal low correlation image in which the phase is adjusted by replacing the elements of the tap position to be
The frequency component reconstruction unit performs Haar wavelet reconstruction using the spatiotemporal low correlation image and the phase-adjusted spatiotemporal low correlation image to generate a super-resolution image of the spatiotemporal low correlation image The spatio-temporal low correlation image super-resolution apparatus according to claim 1, wherein: An image space super-resolution device that generates a super-resolution image of an input image using the spatio-temporal low correlation image super-resolution device according to claim 1 or 2,
A spatiotemporal correlation component separating unit that separates an input image into a spatiotemporal low correlation image composed of pixel components having a low spatiotemporal correlation and a spatiotemporal high correlation image composed of pixel components having a high spatiotemporal correlation;
The spatio-temporal low correlation image super-resolution device that generates a spatio-temporal low correlation super-resolution image by super-resolution processing of the spatio-temporal low correlation image separated by the spatio-temporal correlation component separation unit;
A spatio-temporal highly correlated image super-resolution unit that generates a spatio-temporal highly correlated super-resolution image by super-resolution processing the spatio-temporal highly correlated image separated by the spatio-temporal correlated component separating unit;
An image synthesis unit that generates a super-resolution image by adding and synthesizing the spatio-temporal low-correlation super-resolution image and the spatio-temporal high-correlation super-resolution image;
An image space super-resolution apparatus comprising: The spatio-temporal correlation component separation unit is
Multi-resolution analysis of the input image in the time direction to generate a time low frequency component image that is an image having a low frequency component in the time direction and a time high frequency component image that is an image having a high frequency component in the time direction And
The temporal high-frequency component image is a first-order spatio-temporal low-frequency component image that is an image having a low-frequency component in the spatio-temporal direction by performing multi-resolution analysis in the spatial direction, and an image having a high-frequency component in the spatio-temporal direction. A spatial frequency decomposition unit for generating a primary spatio-temporal high-frequency component image;
The primary spatiotemporal high frequency component image is reconstructed in the spatiotemporal direction to generate the spatiotemporal low correlation image, and the temporal low frequency component image and the primary spatiotemporal low frequency component image are reconstructed in the spatiotemporal direction. A frequency component reconstruction unit configured to generate the spatiotemporal highly correlated image;
The image space super-resolution device according to claim 3, comprising: The spatiotemporal correlation component separation unit calculates an isolated point component level indicating the size of an isolated point component included in the primary spatiotemporal high frequency component image, and the isolated point component level of the primary spatiotemporal high frequency component image is calculated. An image composed of the following pixels is generated as a secondary space- time high-frequency component image, and an image composed of pixels exceeding the isolated point component level is generated as a secondary space-time high-frequency component image. Further comprising a spatiotemporal correlation component detection unit,
The frequency component reconstruction unit reconstructs the secondary spatio-temporal high-frequency component image in a spatio-temporal direction to generate the spatio-temporal low-correlation image, the temporal low-frequency component image, and the primary spatio-temporal low-frequency component. 5. The image space super-resolution device according to claim 4, wherein the image and the secondary spatio-temporal high-frequency component image are reconstructed in a spatio-temporal direction to generate the spatio-temporal highly correlated image.   A spatio-temporal low-correlation image super-resolution program for causing a computer to function as the spatio-temporal low-correlation image super-resolution apparatus according to claim 1 or 2.   An image space super-resolution program for causing a computer to function as the image space super-resolution device according to any one of claims 3 to 5.

Images