JP2017045273A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve image quality of a high-resolution image to be generated from a plurality of low-resolution images.SOLUTION: An alignment coordinate conversion unit (403) aligns a plurality of images other than a reference image, which is one of a plurality of input images and serving as alignment reference, with respect to the reference image. A pixel insertion processing unit (404) inserts pixels of the other images into the reference image. A direction-specific filtering processing unit (406) discriminates an edge direction from the image with the pixels inserted thereto, and performs edge direction-specific filtering on the image with the pixels inserted thereto, on the basis of the discriminated edge direction.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for generating a high resolution image from a plurality of low resolution images.

  As one of the super-resolution techniques for generating a high-resolution image from a low-resolution image, super-resolution that creates a single high-resolution image using multiple low-resolution images (hereinafter, multiple The technology is known. Such a multiple-image super-resolution technique is used in a digital camera or the like when, for example, improving the image quality of an image that is enlarged by electronic zoom.

  For example, a process of super-resolving a plurality of images when a single high-resolution image is generated from a plurality of low-resolution images obtained by an image sensor having a Bayer color filter will be briefly described. From a plurality of low resolution images corresponding to the Bayer array, R (red), G (green), and B (blue) images of the same color component are generated. For each image of the same color component, the pixel value of the color component is arranged at the coordinates corresponding to each pixel of the same color component in the original low resolution image, and the zero value is set at the coordinates corresponding to the pixels of the other color component Are generated by interpolation. Furthermore, a high-resolution image for each color component is generated by enlarging the image for each color component. Thereafter, the pixel value of the corresponding coordinate in another image is inserted into the coordinate where the zero value is interpolated in the high-resolution image for each color component. Then, one super-resolution image is created from the high-resolution image for each color component after such pixel insertion.

  Here, in the multi-pixel super-resolution pixel insertion processing, in order to avoid a decrease in the signal band, the coordinates of the pixel insertion target pixel (zero value interpolation pixel) are set to the coordinates of the pixel insertion image. A process of acquiring the pixel corresponding to the closest coordinate from another image and inserting it as it is is performed.

  However, if a pixel is inserted as it is, for example, if there is a position shift between the pixel on the side where the pixel is inserted and the inserted pixel, an unnatural signal switching occurs at the position shift portion. May be visible. Such an unnatural signal switching portion appears as image quality degradation. Therefore, for an image in which pixel insertion has been performed, it is desired to perform processing that suppresses image quality deterioration due to unnatural signal switching due to pixel position shifts, while minimizing the bandwidth of the image signal.

  Patent Document 1 discloses a technique for generating an interpolated pixel for a low-resolution input image and generating a high-resolution enlarged image by inserting the interpolated pixel into the input image. Further, in Patent Document 1, edge enhancement processing is performed on an enlarged image generated by inserting an interpolation pixel, and different filter processing is performed on the image subjected to edge enhancement processing depending on the edge direction. Techniques to apply are disclosed.

JP 2008-54267 A

  Here, as described above, a high resolution image is generated by pixel insertion in the super-resolution of a plurality of images, and similarly, an enlarged image is generated by insertion of interpolation pixels in the technique described in Patent Document 1. For this reason, in order to cope with the problem in the multi-picture super-resolution processing in which the unnatural signal switching occurs due to the pixel positional deviation as described above, it is considered to apply the technique described in Patent Document 1. It is done.

  However, if the technique described in Patent Document 1 is applied to the multi-image super-resolution processing, an unnatural signal switching portion caused by a positional shift at the time of pixel insertion is caused by edge enhancement processing. It will be further emphasized. For this reason, even if different filter processing according to the edge direction as described in Patent Document 1 is performed, the effect of improving the image quality is hardly obtained. Further, when noise or the like is generated in the image, the noise is further emphasized by the edge enhancement processing, and the image quality may be deteriorated.

  The present invention has been made in view of such problems, and an image processing apparatus, an image processing method, and a program that can improve the image quality of a high-resolution image generated from a plurality of low-resolution images. The purpose is to provide.

  The present invention is an image processing apparatus that generates a high-resolution image using a plurality of low-resolution images, and among the plurality of input images, one image serving as a reference for alignment is used as a reference. A positioning unit that aligns the positions of a plurality of other images excluding the reference image with respect to the reference image, and a pixel by inserting pixels of the plurality of other images with respect to the reference image. A pixel insertion unit that generates an image with a resolution; and a direction of an edge is determined from the image after the pixel is inserted, and the image after the pixel is inserted based on the determined edge direction And filtering means for performing filtering according to the direction of the edge.

  According to the present invention, it is possible to improve the image quality of a high-resolution image generated from a plurality of low-resolution images.

It is a figure which shows schematic structure of the digital camera of this embodiment. It is a figure which shows the structural example of the image process part in the digital camera of this embodiment. It is a flowchart of the whole process of an image processing part. It is explanatory drawing of the position shift which generate | occur | produces in the process of multiple super-resolution. It is explanatory drawing of color component separation of each pixel by a Bayer arrangement. It is a figure which shows the structural example of a coordinate transformation coefficient calculation part. It is a flowchart of the process in a coordinate transformation coefficient calculation part. It is explanatory drawing of the direction discrimination | determination by a 3-tap filter. It is a figure which shows the structural example of a super-resolution process part. It is a flowchart of the process in a super-resolution process part. It is explanatory drawing of coordinate transformation. It is explanatory drawing of pixel insertion in the case of the production | generation of a super-resolution image. It is a flowchart of the process in a pixel insertion process part. It is a figure used for description of a reference coordinate. It is explanatory drawing of the ratio of the non-missing pixel and a missing pixel in a super-resolution image. It is a flowchart of the process in the filtering process part according to direction. It is explanatory drawing of the number of initial taps. It is explanatory drawing of the direction discrimination | determination by a 5-tap filter. It is explanatory drawing of the total process of a direction discrimination | determination result. It is explanatory drawing of the filter coefficient calculation method of the filtering process according to direction.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The image processing apparatus according to the present embodiment is not limited to an imaging apparatus such as a digital camera, but may be various information processing apparatuses such as a tablet terminal, a smartphone, an electronic game machine, a drive recorder, a navigation apparatus, and a personal computer. In this case, the information processing apparatus acquires a photographed image photographed by a camera function mounted on the apparatus itself or a photographed image photographed by a digital camera or the like, and this embodiment described later on the photographed image. Each process according to the form is executed. Further, the processing according to the present embodiment in the information processing apparatus may be realized by executing a computer program in an internal CPU or the like. A computer program for realizing the processing according to the present embodiment is provided to the information processing apparatus via a recording medium, various networks, or a communication line.
The image processing apparatus according to the present embodiment uses a plurality of low-resolution images, generates a high-resolution image in which the number of pixels is expanded k times in the horizontal and vertical directions with respect to the number of pixels of the low-resolution image, Perform super-resolution processing to achieve resolution.

FIG. 1 shows a schematic configuration of a digital camera to which the image processing apparatus of this embodiment is applied.
In FIG. 1, an optical system 101 includes a lens group including a zoom lens, a focus lens, and the like, an aperture adjustment mechanism, a shutter mechanism, and the like. The optical system 101 forms a subject image or the like on the imaging surface of the image sensor 102, and the magnification, focus position, light amount, etc. of the subject image when the subject image is imaged on the imaging surface of the image sensor 102. Adjust.

  The imaging element 102 is a photoelectric conversion element such as a CCD or CMOS sensor that photoelectrically converts an optical image such as a subject image formed on the imaging surface by the optical system 101 into an electrical signal. Further, the image sensor 102 includes R (red), G (green), and blue (B) primary color filters arranged in a so-called Bayer array on the imaging surface, so that a color image can be captured. . The A / D conversion unit 103 converts the analog imaging signal input from the imaging element 102 into digital image data. The image data output from the A / D conversion unit 103 is sent to the image processing unit 104.

  In addition to normal signal processing such as white balance correction processing, distortion correction, noise reduction processing, and development processing, the image processing unit 104 performs super-resolution processing of this embodiment using a plurality of input images. . The image processing unit 104 can perform the same super-resolution processing not only on the image output from the A / D conversion unit 103 but also on the image read out from the recording unit 108. Details of the super-resolution processing performed in the image processing unit 104 will be described later.

  The operation unit 106 includes various buttons, keys, a dial, a touch panel, and the like. When these are operated by a user, instruction information generated by the user operation is generated and output to the system control unit 105. The display unit 107 includes a liquid crystal display, an organic EL (Electro Luminescence) display, and the like, and displays an image captured by the image sensor 102 and developed by the image processing unit 104, an image read from the recording unit 108, and the like. . The recording unit 108 records, for example, image data and various information on an information recording medium using a package containing a rotating recording body such as a memory card mounted with a semiconductor memory or a magneto-optical disk. And read information. The recording unit 108 may have a function of detachably attaching these information recording media. The information recording medium may store a program in addition to image data and various types of information. A bus 109 is used to exchange image data among the image processing unit 104, the system control unit 105, the display unit 107, and the recording unit 108.

  The system control unit 105 is a control function unit that controls and controls the overall operation of the image processing apparatus according to the present embodiment. The system control unit 105 also performs drive control of the optical system 101 and the image sensor 102 based on the luminance value obtained from the image processed by the image processing unit 104 and user instruction information input via the operation unit 106. . The system control unit 105 also performs display control of the display unit 107 and writing and reading control of the recording unit 108.

Hereinafter, a specific configuration and processing flow related to the super-resolution processing performed in the image processing unit 104 will be described. FIG. 2 shows a specific configuration example related to the super-resolution processing performed by the image processing unit 104.
As shown in FIG. 2, the image processing unit 104 includes a reference image selection unit 201, a Bayer separation unit 202, a coordinate conversion coefficient calculation unit 203, a memory unit 204, a super-resolution process, as main components related to the super-resolution processing. Unit 205 and YUV conversion unit 206.
FIG. 3 shows a flowchart of processing performed by the image processing unit 104 having the configuration shown in FIG. The operation of each part in the configuration shown in FIG. 2 will be described below with reference to the flowchart of FIG.

  In step S501, the image processing unit 104 acquires images sequentially input from the A / D conversion unit 103 in FIG. 1 or acquires images sequentially read from the recording unit 108 in FIG. Note that the image input to the image processing unit 104 at this time is an image including pixels corresponding to the Bayer array of the color filter described above, and is hereinafter referred to as a “Bayer image”. In step S501, the image processing unit 104 sequentially acquires N Bayer images. After step S501, the image processing unit 104 shifts the processing to step S502 performed by the reference image selection unit 201.

  In step S502, the reference image selection unit 201 selects an image serving as a reference for performing super-resolution processing from among a plurality (N) of Bayer images sequentially acquired in step S501. In the case of the present embodiment, the reference image selection unit 201 selects, for example, the first image among the plurality of Bayer images acquired in order as a reference image. Note that an image serving as a reference in performing super-resolution processing refers to an image serving as a reference for alignment described later. In the following description, an image serving as a reference for alignment is referred to as “reference image”, and other images are referred to as “non-reference images”.

  Although details will be described later, in this embodiment, a super-resolution image is generated by inserting pixels of a non-reference image into a reference image. Here, for example, as illustrated in FIG. 4, the pixel 2003 is a pixel into which no pixel is inserted, and the pixel 2004 is a pixel into which a pixel from another image is inserted. For example, it is assumed that there is a positional deviation D between the center coordinate 2001 of the pixel 2004 and the center coordinate 2002 of the pixel inserted with respect to the pixel 2004. When such a positional deviation D occurs, an unnatural signal switching occurs between the pixel 2003 and the pixel 2004 after the pixel insertion, and this appears as image quality degradation. Therefore, in the present embodiment, the non-reference image is aligned with the reference image in consideration of the positional deviation D when the pixels are inserted.

After step S502, the reference image selection unit 201 advances the process to step S503.
In step S503, the reference image selection unit 201 selects a reference image from the plurality of Bayer images described above, outputs the reference image first, and then outputs each non-reference image in the order of input. In this example, among the N Bayer images acquired in step S501, the first image is output as a reference image, and the second to Nth images are output as non-reference images in the order of input. Is done. Further, when the reference image selection unit 201 sequentially outputs the images, the reference image selection unit 201 sets a variable M that represents the number of images among the N Bayer images. For example, when the reference image is output, the variable M is set to “1”. After step S503, the image processing unit 104 shifts the processing to step S504 performed by the Bayer separation unit 202.

  In step S504, the Bayer separation unit 202 separates the Mth Bayer image output from the reference image selection unit 201 in step S503 for each color component of R (red), G (green), and B (blue). Then, three mosaic images for each same color component are generated. When the variable M is “1”, the Bayer separation unit 202 generates three mosaic images for each of R, G, and B color components from the reference image of the Bayer image.

Hereinafter, the Bayer image color component separation processing performed by the Bayer separation unit 202 will be described with reference to FIG. 5.
FIG. 5A illustrates an example of a Bayer image output from the reference image selection unit 201 and having the horizontal pixel count “W” and the vertical pixel count “H”. FIGS. 5B to 5D show mosaic images obtained by separating the Bayer image of FIG. 5A for each of the R, G, and B color components. In the mosaic image for each color component, the pixel value of the color component is arranged at the coordinates corresponding to each pixel of the same color component in the original Bayer image, and zero (0) is set at the coordinates corresponding to the pixels of other color components. ) Value pixels are generated by interpolation. Specifically, FIG. 5B shows a mosaic image corresponding to the R color component. Hereinafter, the R color component is referred to as “R component”. The mosaic image in FIG. 5B includes the non-missing pixel 901R having information on each pixel value corresponding to the R component of the Bayer image in FIG. 5A and the pixel value information in the Bayer image in FIG. And a missing pixel 902 in which a zero (0) value is inserted. Similarly, the mosaic images in FIGS. 5C and 5D are non-missing pixels 901G and 901B having information on pixel values corresponding to the G and B color components of the Bayer image in FIG. 5A, respectively. And a missing pixel 902 in which a zero value is inserted. Hereinafter, the G color component is referred to as “G component”, and the B color component is referred to as “B component”.

  The data of the three mosaic images for each color component of R, G, and B generated by the Bayer separation processing as described in FIGS. 5A to 5D is the super-resolution processing unit 205. To the coordinate conversion coefficient calculation unit 203 and the memory unit 204. After step S504, the image processing unit 104 proceeds to step S507 through steps S505 and S506 described later.

  In step S507, the image processing unit 104 determines whether the variable M is “N”, which is the number of acquired Bayer images. If it is determined in step S507 that M is not N (M ≠ N), the image processing unit 104 proceeds to step S508, increments the variable M (increases by “1”), and proceeds to step S508. The process returns to S504. If the image processing unit 104 determines in step S507 that M has become N (M = N), the image processing unit 104 proceeds to step S509.

  When the variable M is incremented in step S508 and the process returns to step S504, the Bayer separation unit 202 performs Bayer separation processing on the next Bayer image. Thereafter, the image processing unit 104 performs processing in steps S505 and S506, which will be described later, and then proceeds to step S507. In this manner, the image processing unit 104 repeats the processing from steps S504 to S507 until the variable M becomes the number N of acquired Bayer images by the increment in step S508. That is, the image processing unit 104 performs the processing from step S504 to step S507 from the first (M = 1) Bayer image (reference image) to the second (M = 2) to Nth (M =). N) Each Bayer image (non-reference image) is sequentially performed.

  The process of step S505 is performed by the coordinate conversion coefficient calculation unit 203. In step S505, the coordinate conversion coefficient calculation unit 203 calculates a coordinate conversion coefficient necessary for positioning each non-reference image with respect to the reference image. In the case of the present embodiment, the coordinate conversion coefficient calculation unit 203 performs a demosaic process described later on the mosaic image of the G component, performs a distortion correction process on the image after the demosaic process, and performs the distortion correction process. An image for G component alignment is created from the image. Hereinafter, the G component alignment image is referred to as “alignment G image”. When the variable M is “1”, the coordinate conversion coefficient calculation unit 203 stores the G image for alignment of the reference image in the memory unit 204, and then each time the variable M is incremented in step S508, the position of each non-reference image. The alignment G images are generated in order and stored in the memory unit 204. Then, the coordinate conversion coefficient calculation unit 203 uses the reference G image for alignment of the reference image and the alignment G image for each non-reference image generated in turn thereafter, and uses the reference image and each of the non-reference images. Projective transformation coefficients for positioning each of the two are calculated.

FIG. 6 shows a detailed configuration of the coordinate conversion coefficient calculation unit 203 in FIG. 2 that performs the coordinate conversion coefficient calculation processing in step S505 in FIG.
As shown in FIG. 6, the coordinate transformation coefficient calculation unit 203 includes a demosaic processing unit 301, a distortion correction processing unit 302, a projective transformation coefficient calculation unit 303, and a direction determination processing unit 304.
FIG. 7 shows a flowchart of processing performed by the coordinate conversion coefficient calculation unit 203 shown in FIG. The operation of each part in the configuration shown in FIG. 6 will be described below with reference to the flowchart of FIG.

  In step S601, the coordinate conversion coefficient calculation unit 203 acquires the Mth mosaic image of the G component input from the Bayer separation unit 202 described above. After step S601, the coordinate conversion coefficient calculation unit 203 shifts the processing to step S602 performed in the demosaic processing unit 301.

  In step S602, the demosaic processing unit 301 performs demosaic processing on the Mth mosaic image of the G component acquired in step S601. As the demosaic process, a known process such as a linear interpolation process or an adaptive interpolation process is used. Detailed description of linear interpolation processing and adaptive interpolation processing will be omitted. After step S602, the coordinate conversion coefficient calculation unit 203 shifts the processing to step S603 performed in the distortion correction processing unit 302.

  In step S603, the distortion correction processing unit 302 performs coordinate correction processing for distortion correction processing on the G component image that has been demosaiced in step S602. As the distortion correction processing, known processing such as correction by image deformation based on lens information representing distortion aberration characteristics of the lens of the optical system 101 in FIG. 1 can be used, and detailed description thereof is omitted. The lens information representing the distortion characteristic is information representing the shift amount of each of R, G, and B color components based on the image height from the center of the optical axis of the lens of the optical system 101. The lens information indicating the distortion characteristic is stored in, for example, a memory (not shown in FIG. 1) included in the optical system 101, the recording unit 108, and the like, and is read out by the system control unit 105 to perform image processing. The distortion correction processing unit 302 of the unit 104 is supplied. After step S602, the coordinate conversion coefficient calculation unit 203 advances the process to step S604.

  In step S604, the coordinate conversion coefficient calculation unit 203 determines whether the G component image subjected to the distortion correction process in step S603 is an image generated from the first (M = 1) reference image. If the coordinate conversion coefficient calculation unit 203 determines in step S604 that the image is generated from the first (M = 1) reference image, the process proceeds to step S611. The processing in step S611 is processing performed in the direction determination processing unit 304. On the other hand, the coordinate conversion coefficient calculation unit 203 determines in step S604 that it is not an image generated from the first (M = 1) reference image, that is, an image generated from a non-reference image. In the case, the process proceeds to step S605. The process of step S605 is a process performed by the projective transformation coefficient calculation unit 303.

  When the processing proceeds to step S611, the direction determination processing unit 304 sets the first component (reference image) G component image after the distortion correction processing in step S603 as a “reference alignment G image”. Further, the direction determination processing unit 304 performs edge direction determination processing for all the pixels of the reference alignment G image. The direction determination processing performed by the direction determination processing unit 304 is the same as the direction determination processing performed by the projection conversion coefficient calculation unit 303 described later in step S609. Details of the direction determination processing performed by the projection conversion coefficient calculation unit 303 are as follows. It will be described later. After step S611, the direction determination processing unit 304 proceeds with the process to step S612.

  In step S <b> 612, the direction determination processing unit 304 stores the reference alignment G image and direction determination result information obtained by the direction determination processing in the memory unit 204. After step S612, the coordinate conversion coefficient calculation unit 203 ends the process of the flowchart of FIG. Thereby, the image processing unit 104 advances the processing to step S506 in FIG.

  On the other hand, when the process proceeds to step S605 because the variable M is 2 or more in step S604, the projective transformation coefficient calculation unit 303 determines that the G component image corrected in distortion in step S603 is “G image for non-reference alignment”. " Further, the projective transformation coefficient calculation unit 303 at this time acquires a reference alignment G image from the memory unit 204. After step S605, the projective transformation coefficient calculation unit 303 advances the process to step S606.

  In step S606, the projective transformation coefficient calculation unit 303 detects a positional shift amount between the non-reference alignment G image and the reference alignment G image obtained in step S605. Specifically, the projective transformation coefficient calculation unit 303 detects a positional deviation amount of the non-reference alignment G image corresponding to the reference alignment G image at a plurality of locations in the image. As the positional deviation amount detection process, a known process such as a process using pattern matching can be used, and a detailed description thereof will be omitted. After step S606, the projective transformation coefficient calculation unit 303 advances the process to step S607.

In step S607, the projection conversion coefficient calculation unit 303 calculates a coordinate conversion coefficient from the positional deviation amount detected in step S606. In the case of the present embodiment, the projection conversion coefficient calculation unit 303 calculates the coefficients a _{0 to} a ₇ of the projection conversion formula shown in the following formula (1) as coordinate conversion coefficients. The coefficients a _{0 to} a ₇ can be calculated using a known method such as a least square method, for example, and a detailed description thereof will be omitted. Note that (xp, yp) in the equation (1) is a coordinate after the projective transformation of the coordinate (x, y). After step S607, the projective transformation coefficient calculation unit 303 advances the process to step S608.

xp = (a ₀ x + a ₁ y + a ₂ ) / (a ₆ x + a ₇ y + 1)
yp = (a ₃ x + a ₄ y + a ₅ ) / (a ₆ x + a ₇ y + 1) (1)

In step S608, the projective transformation coefficient calculation unit 303 transforms the non-reference alignment G image based on the coordinate transformation coefficients a _{0 to} a ₇ calculated in step S607. Thereby, the position between the non-reference alignment G image and the reference alignment G image is adjusted. In addition, as a deformation | transformation process, a well-known bilinear interpolation process etc. can be used, The detailed description is abbreviate | omitted. After step S608, the projective transformation coefficient calculation unit 303 advances the process to step S609.

  In step S609, the projective transformation coefficient calculation unit 303 performs edge direction determination processing on all pixels of the alignment G image of the Mth non-reference image deformed in step S608.

Hereinafter, the direction determination process performed by the projective transformation coefficient calculation unit 303 in step S609 will be described with reference to FIGS. 8A to 8D.
In the present embodiment, the direction determination process performed by the projective transformation coefficient calculation unit 303 is, for example, as shown in FIG. 8A to FIG. The process is to determine four directions from × 3 pixels. Here, among the 3 × 3 surrounding pixels, a pixel that is a target of the direction determination process with respect to the target pixel 1601 is referred to as a “target pixel”. The projective transformation coefficient calculation unit 303 includes the direction of the target pixels 1602 and 1603 in FIG. 8A, the direction of the target pixels 1604 and 1605 in FIG. 8B, the direction of the target pixels 1606 and 1607 in FIG. The direction of the target pixels 1608 and 1609 in FIG.

  Specifically, assuming that the pixel value of the pixel of interest 1601 is “pix” and the values of the target pixel are pix1 and pix2, the projective transformation coefficient calculation unit 303 calculates the difference absolute value sum sad as shown in the following equation (2). The calculation process is performed for each of the four directions shown in FIGS. 8 (a) to 8 (d). Note that the values “pix1” and “pix2” of the target pixel are used as “pixi” in Expression (2).

  Then, the projective transformation coefficient calculation unit 303 calculates the smallest absolute difference among the sum of absolute differences sad1, sad2, sad3, and sad4 calculated for the four directions shown in FIGS. 8 (a) to 8 (d). The direction in which the value sum min_sad is calculated is taken as a direction discrimination result. That is, the direction in which this minimum difference absolute value sum min_sad is calculated represents the direction of the edge. However, when the difference between the minimum difference absolute value sum min_sad and each of the other difference absolute value sums sad is equal to or less than a predetermined threshold set in advance for comparison with the difference, for example, projective transformation The coefficient calculation unit 303 determines that there is almost no difference between the pixel of interest and its surrounding pixels. In this case, the projective transformation coefficient calculation unit 303 outputs an error as the direction determination result, assuming that the edge direction is not detected in all four directions. Note that the above-described predetermined threshold value for the difference between the minimum difference absolute value sum min_sad and each other difference absolute value sum sad may be changed as an arbitrary parameter, for example, for each scene of the photographed image.

Returning to FIG.
After step S609 in FIG. 7, the projective transformation coefficient calculation unit 303 advances the process to step S610. In step S610, the projective transformation coefficient calculation unit 303 sets the coordinate transformation coefficient calculated in step S607 as the coordinate transformation coefficient for the M-th non-reference image, and uses the direction discrimination result calculated in step S608 as the M-th non-reference image. And stored in the memory unit 204. The coordinate conversion coefficient and the direction determination result are used in the super-resolution processing unit 205 later. Information on the coordinate transformation coefficient and the direction discrimination result for the M-th non-reference image stored in the memory unit 204 is used later in the super-resolution processing unit 205. After step S610, the coordinate conversion coefficient calculation unit 203 ends the process of the flowchart of FIG.
The above is the process flow of step S505 in FIG. 3 in the coordinate conversion coefficient calculation unit 203.

Returning to FIG.
After the processing in step S505 in FIG. 3, the image processing unit 104 shifts the processing to step S506 performed in the super-resolution processing unit 205 in FIG.
In step S506, the super-resolution processing unit 205 performs super-resolution processing for creating a high-resolution image using the low-resolution image. In the case of the present embodiment, the super-resolution processing unit 205 performs a mosaic image for each R, G, B color component after the Bayer separation process of the reference image and an R, G, B after the Bayer separation process of the non-reference image. Super-resolution processing is performed using a mosaic image for each color component. Details of the super-resolution processing in the super-resolution processing unit 205 will be described later.

  After step S506, the image processing unit 104 advances the process to step 507 described above. If it is determined in step S507 that the variable M has reached the number of acquired Bayer images N (M = N), the image processing unit 104 performs processing in step S509 performed by the YUV conversion unit 206 in FIG. To migrate.

  In step S509, the YUV conversion unit 206 creates a YUV image from the super-resolution image for each color component of R, G, B generated by the super-resolution processing of the super-resolution processing unit 205. . That is, this YUV image is a high-resolution super-resolution image created from the super-resolution image of each color component of R, G, B. Expression (3) shows a data conversion expression when the YUV conversion unit 206 generates YUV image data from image data of R, G, and B color components.

Y = 0.299R + 0.587G + 0.114B
U = −0.169R−0.331G + 0.5B Formula (3)
V = 0.5R-0.419G-0.081B

  When the image processing unit 104 outputs the YUV image data created in step S509, the image processing unit 104 ends the process of the flowchart of FIG.

Hereinafter, the details of the super-resolution processing performed by the super-resolution processing unit 205 in FIG. 2 in step S506 of the flowchart in FIG. 3 will be described.
In the present embodiment, an example in which a super-resolution image is generated from a high resolution obtained by enlarging the number of pixels of a low-resolution image for each color component of R, G, and B by k times in the horizontal and vertical directions. Super-resolution processing by the resolution processing unit 205 will be described.

  In the present embodiment, since the high-resolution image is generated by expanding the number of pixels of the low-resolution image by k times in the horizontal and vertical directions, the super-resolution processing unit 205 determines the coordinates and the high-resolution of each pixel of the low-resolution image. A coordinate conversion process for associating the coordinates of each pixel of the image is performed. During this coordinate conversion processing, the super-resolution processing unit 205 corresponds to the coordinate conversion for distortion correction and the coordinates of each pixel between the low-resolution image and the high-resolution image for the G component low-resolution image. And coordinate conversion for alignment. In the coordinate conversion process, the super-resolution processing unit 205 performs coordinate conversion for correcting the lateral chromatic aberration, coordinate conversion for correcting the distortion, low-resolution image, and high-resolution image for the R and B component low-resolution images. Coordinate conversion is performed for alignment in which the coordinates of each pixel correspond to the resolution image. Then, the super-resolution processing unit 205 converts the pixel value of each coordinate after the coordinate conversion processing on the low-resolution image for each color component of R, G, B into the high-resolution image for each color component of R, G, B. A process of inserting the pixel value of the corresponding coordinate is performed. After such pixel values are inserted, the super-resolution processing unit 205 performs missing pixel interpolation processing and edge direction as described later on the high-resolution image for each of R, G, and B components. Perform another filtering process.

FIG. 9 shows a detailed configuration example of the super-resolution processing unit 205 in FIG.
As shown in FIG. 9, the super-resolution processing unit 205 includes a chromatic aberration coordinate correction unit 401, a distortion coordinate correction unit 402, a coordinate conversion unit 403 for alignment, a pixel insertion processing unit 404, a missing interpolation processing unit 405, and a direction-specific operation. A filtering processing unit 406 is included.
FIG. 10 shows a flowchart of processing performed by each unit of the super-resolution processing unit 205 shown in FIG. The operation of each part in the configuration shown in FIG. 9 will be described below with reference to the flowchart of FIG.

  In the flowchart of FIG. 10, the processing from step S701 to step S706 is processing related only to the coordinates of each pixel of each color component of R, G, B. For this reason, the super-resolution processing unit 205 first initializes the coordinates of each mosaic image, which is a low-resolution image for each color component of R, G, and B, to (0, 0) as the process of step S701. In the present embodiment, the coordinates (0, 0) initialized in the mosaic image for each color component of R, G, B represent, for example, the coordinates of the center of the upper left pixel as shown in FIG. Yes. As shown in FIG. 11A, the right direction is the x direction and the downward direction is the y direction from the coordinates (0, 0). Accordingly, the coordinates of each pixel in the mosaic image for each color component of R, G, and B are represented by coordinates R (x, y), G (x, y), and B (x, y), respectively. After step S701, the super-resolution processing unit 205 shifts the processing to step S702 performed by the magnification chromatic aberration coordinate correction unit 401.

  In step S702, the magnification chromatic aberration coordinate correction unit 401 corrects the magnification chromatic aberration with respect to the coordinates R (x, y) and coordinates B (x, y) of each pixel of the R component and B component mosaic images. Coordinate correction processing is performed. In the present embodiment, the lateral chromatic aberration coordinate correction unit 401 performs coordinate correction processing for correcting lateral chromatic aberration based on lens information representing the lateral chromatic aberration characteristics of the lens of the optical system 101 in FIG. The lens information representing the magnification chromatic aberration characteristic is information representing each deviation amount of the R and B components with respect to the G component based on the image height from the center of the optical axis of the lens of the optical system 101. And the information for performing the coordinate correction only for the B component. The lens information representing the magnification chromatic aberration characteristic is stored in, for example, a memory (not shown in FIG. 1) provided in the optical system 101, the recording unit 108, and the like, and is read by the system control unit 105. And is supplied to the super-resolution processing unit 205 of the image processing unit 104.

  FIG. 11B and FIG. 11D show the coordinates of the R component and the B component before and after the coordinate correction processing for correcting the chromatic aberration of magnification in step S702. The white circles in FIG. 11B represent the coordinates R (x, y) of each pixel of the R component before the coordinate correction process for correcting the lateral chromatic aberration, and the black circles in the figure represent the respective coordinates after the coordinate correction process. The coordinates R (xr, yr) are represented. Similarly, the white circles in FIG. 11D represent the coordinates B (x, y) of each pixel of the B component before the coordinate correction process for correcting the lateral chromatic aberration, and the black circles in the figure represent the coordinate correction process. Each subsequent coordinate B (xb, yb) is represented. Note that FIG. 11C shows the coordinates G (x, y) of each pixel of the mosaic image of the G component. For the G component, the coordinate correction processing by the magnification chromatic aberration coordinate correction unit 401 is not performed, and thus the coordinates after the coordinate correction processing for magnification chromatic aberration correction are not drawn in FIG. After step S702, the super-resolution processing unit 205 shifts the processing to step S703 performed in the distortion coordinate correction unit 402.

  In step S703, the distortion coordinate correction unit 402 performs the coordinate G (x, y) of each pixel of the G component and the coordinate R (xr, yr) of each pixel of the R and B components after the coordinate correction processing of the magnification chromatic aberration correction. , B (xb, yb) are subjected to coordinate correction processing for distortion correction. In the present embodiment, the distortion coordinate correction unit 402 performs coordinate correction processing for correcting distortion chromatic aberration based on lens information representing distortion aberration characteristics of the lens of the optical system 101. The lens information representing the distortion characteristic is information representing the shift amount of each color component of R, G, B based on the image height from the center of the optical axis of the lens of the optical system 101, and coordinates for correcting distortion aberration. It is information for performing correction processing for each of the R, G, and B color components. The lens information representing the distortion aberration characteristic is stored in, for example, a memory of the optical system 101, and read by the system control unit 105 and super-resolution processing of the image processing unit 104, like the lens information of the lateral chromatic aberration characteristic. Supplied to the unit 205.

  FIGS. 11E to 11G show the coordinates of the R, G, and B color components before and after the coordinate correction process for correcting distortion aberration in step S703. The white circles in FIG. 11 (e) represent the coordinates R (xr, yr) of each pixel of the R component before the coordinate correction process for distortion correction, and the black circles in the figure represent the coordinates after the coordinate correction process. R (xrd, yrd) is represented. Similarly, white circles in FIG. 11G represent the coordinates B (xb, yb) of each B component pixel before coordinate correction processing for distortion correction, and black circles in the drawing indicate coordinate correction processing. Each coordinate B (xbd, ybd) after being made is represented. Also, the white circles in FIG. 11 (f) represent the coordinates G (x, y) of each pixel of the G component before the coordinate correction process for distortion correction, and the black circles in the figure after the coordinate correction process. Each coordinate G (xd, yd) is represented. After step S703, the super-resolution processing unit 205 shifts the process to step S704 performed in the alignment coordinate conversion unit 403.

  In step S704, the coordinate conversion unit 403 for alignment uses the first image (reference image) as the mosaic image for each color component of R, G, and B input to the super-resolution processing unit 205 at this time. It is determined whether or not the mosaic image is generated from. If the coordinate conversion unit for alignment 403 determines in step S704 that the mosaic image is generated from the first reference image, the process proceeds to step S705. On the other hand, if the coordinate conversion unit for alignment 403 determines in step S704 that the mosaic image is generated from a non-reference image that is not the first image, the process proceeds to step S706.

  In step S705, the coordinate conversion unit for alignment 403 performs coordinate enlargement processing on the coordinates of the R, G, and B color components after the coordinate correction processing by the distortion coordinate correction unit 402. That is, in step S705, coordinate enlargement processing is performed on the coordinates of the R, G, and B color components corresponding to the first reference image. In the case of the present embodiment, the alignment coordinate conversion unit 403 applies the coordinates R (xrd, yrd), G (xd, yd), and B (xbd, ybd) after the coordinate correction processing by the distortion coordinate correction unit 402. The values of x and y of the coordinates are each expanded k times by the calculation of the following equation (4). Note that k in Equation (4) is a parameter that represents an enlargement ratio when generating a high-resolution image.

R (xrd, yrd) · k = R (xrd · k, yrd · k)
G (xd, yd) · k = G (xd · k, yd · k) (4)
B (xbd, ybd) · k = B (xbd · k, ybd · k)

  FIGS. 11H to 11J show the coordinates before and after the coordinate enlargement process by the alignment coordinate conversion unit 403 in the process of step S705. In FIG. 11H, white circles represent the coordinates R (xrd, yrd) of the R component before the coordinate enlargement process, and black circles in the figure represent the coordinates R (xrd · k, yrd · y) after the coordinate enlargement process. k). The white circles in FIG. 11 (i) represent the coordinates G (xd, yd) of the G component before the coordinate enlargement process, and the black circles in the figure represent the coordinates G (xd · k, yd ·) after the coordinate enlargement process. k). Similarly, white circles in FIG. 11J represent the coordinates B (xbd, ybd) of each pixel of the B component before the coordinate enlargement process, and black circles in the figure represent the coordinates B (xbd) after the coordinate enlargement process. K, ybd · k).

  Hereinafter, the coordinate values after coordinate enlargement processing for the coordinates of the R, G, and B color components of the reference image performed in step S705 are represented as coordinates R (x ′, y ′), G (x ′, y ′), Indicated as B (x ′, y ′). After step S705, the super-resolution processing unit 205 shifts the processing to step S707 performed in the pixel insertion processing unit 404.

  On the other hand, when the process proceeds to step S706, the alignment coordinate conversion unit 403 performs projective conversion processing on the coordinates of the R, G, and B color components after the coordinate correction processing by the distortion coordinate correction unit 402. Perform coordinate enlargement processing. Specifically, the alignment coordinate conversion unit 403 reads out the coordinate conversion coefficient obtained by the coordinate conversion coefficient calculation unit 203 in FIG. 2 and stored in the memory unit 204, and uses the coordinate conversion coefficient to read the non-reference image. Projective transformation processing is performed on the coordinates of the R, G, and B color components. As a result, the coordinates of the R, G, B color components of the non-reference image are matched with the coordinates of the R, G, B color components of the reference image. Then, the alignment coordinate conversion unit 403 performs coordinate expansion processing for expanding the x value and y value of the coordinate by k times with respect to the coordinates of the R, G, and B color components after the projective conversion processing. .

  In this embodiment, a high-resolution image in which the number of pixels of the low-resolution image is expanded k times in both the horizontal and vertical directions is generated. Therefore, the coordinate conversion unit 403 for alignment uses the projection conversion formula of the above-described formula (1). On the other hand, the calculation of Expression (5) is performed to further expand the coordinates k times in both the horizontal and vertical directions.

x ′ = k · (a ₀ x + a ₁ y + a ₂ ) / (a ₆ x + a ₇ y + 1)
y ′ = k · (a ₃ x + a ₄ y + a ₅ ) / (a ₆ x + a ₇ y + 1) (5)

In Equation (5), the coefficients a _{0 to} a ₇ are the same as those used in Equation (1) described above. In addition, the coordinates (x, y) values input to Equation (5) are the coordinates R (xrd, yrd), G (xd, yd), B () obtained by the distortion coordinate correction unit 402 in step S703. xbd, ybd).

  Thereafter, the coordinate values after the projective transformation process and the coordinate enlargement process for the coordinates of the R, G, and B color components of the non-reference image performed in step S706 are set to the coordinates R (x ', Y'), G (x ', y'), and B (x ', y'). After step S706, the super-resolution processing unit 205 shifts the processing to step S707 performed in the pixel insertion processing unit 404.

  In step S707, the pixel insertion processing unit 404 reads each pixel value of the mosaic image for each color component of R, G, B stored in the memory unit 204 of FIG. Specifically, the pixel insertion processing unit 404 performs coordinate R (x, y), G (x, y), B (x) before the coordinate correction process and the coordinate conversion process from step S702 to step S707 are performed. , Y), each pixel value is read out. In the following description, pixel values corresponding to these coordinates R (x, y), G (x, y), and B (x, y) are expressed as Rpix, Gpix, and Bpix. After step S707, the pixel insertion processing unit 404 advances the process to step S708.

  In step S708, the pixel insertion processing unit 404, the pixel values Rpix, Gpix, Bpix, and the above-described coordinates R (x ′, y ′), G (x ′, y ′) of the reference image and the non-reference image. , B (x ′, y ′), pixel insertion processing is performed for each of the R, G, B color components.

Hereinafter, the concept of the pixel insertion process in the super-resolution process will be described with reference to FIGS. 12 (a) and 12 (b).
FIGS. 12A and 12B are diagrams for explaining the concept of pixel insertion, taking the G component as an example. Here, for example, using a plurality of low-resolution images having the number of pixels in the horizontal direction W and the number of images in the vertical direction H, a super-resolution image in which the number of pixels is doubled in the horizontal and vertical directions is generated. At this time, an example in which pixel insertion processing is performed on the high resolution image 10a as shown in FIG.

  A high-resolution image 10a shown in FIG. 12A is a double-magnification of a low-resolution image having the number of pixels W in the horizontal direction and the number of pixels H in the vertical direction, and the number of pixels in the horizontal direction is 2W and the number of pixels in the vertical direction. This is an image of 2H, and serves as a foundation for generating a super-resolution image. Here, in order to make the explanation easy to understand, in the high resolution image 10a in the example of FIG. 12A, it is assumed that all pixel values are set to zero values that mean missing pixels. The pixel insertion process in the super-resolution process is a process for inserting the value of the non-missing pixel 1001 of the low resolution image into the missing pixel 1002 of the high resolution image 10a. By performing such pixel insertion processing using a plurality of low-resolution images, a super-resolution image 10b as shown in FIG. 12B can be generated.

  In the case of the present embodiment, the pixel insertion processing unit 404 in FIG. 9 generates a high-resolution image from a low-resolution reference image, and detects missing pixels of the high-resolution image of the reference image as a plurality of non-reference images that are low-resolution images. A pixel insertion process is performed in which the pixel value of the non-missing pixel of the image is replaced.

FIG. 13 shows a flowchart of the pixel insertion processing performed by the pixel insertion processing unit 404 in FIG. 9 in step S708 in FIG.
Hereinafter, the pixel insertion processing by the pixel insertion processing unit 404 will be described in detail with reference to the flowchart of FIG. In FIG. 13, the pixel insertion process is described by taking the G component as an example, but the same process is performed for the R and B components, and the process related to the G component in FIG. Since this can be realized by replacing the processing, the description of the R and B component pixel insertion processing is omitted.

  First, in step S801, the pixel insertion processing unit 404 uses, as reference coordinates, coordinates that are closest to the G component coordinate coordinates G (x ′, y ′) obtained from the reference image and can be represented by integers. Calculate as G0 (x0, y0). FIG. 14A is a diagram illustrating an example of the coordinates G (x ′, y ′) and the reference coordinates G0 (x0, y0) in the G component high-resolution image 1100. FIG. 14B is an enlarged view of one pixel extracted from the coordinates G (x ′, y ′) and the reference coordinates G0 (x0, y0) in FIG. The reference coordinates G0 (x0, y0) are calculated by the following equation (6). After step S801, the pixel insertion processing unit 404 advances the process to step S802.

  In step S802, the pixel insertion processing unit 404 determines whether or not the above-described pixel value Gpix is not a zero value (0) (is greater than zero). If the pixel insertion processing unit 404 determines in step S802 that the pixel value Gpix is not zero, the pixel insertion processing unit 404 determines that the pixel having the pixel value Gpix is a non-missing pixel, and proceeds to step S803. On the other hand, if the pixel insertion processing unit 404 determines in step S802 that the pixel value Gpix is a zero value, the pixel insertion processing unit 404 determines that the pixel having the pixel value Gpix is a missing pixel, and proceeds to step S804.

  When the processing proceeds from step S802 to step S803, the pixel insertion processing unit 404, as shown in FIG. 14B, the distance between the coordinates G (x ′, y ′) and the reference coordinates G0 (x0, y0). d (G) is calculated. This distance d (G) is used in subsequent determinations in steps S808 and S810.

Hereinafter, the distance d (G) will be described with reference to FIGS. 12 (a) and 12 (b).
A reference coordinate G0 (x0, y0) that is closest to the coordinate G (x ′, y ′) and is represented by an integer is a coordinate that indicates the center of a certain pixel of the high-resolution image 1100. Therefore, in the present embodiment, the pixel insertion processing unit 404 has coordinates G (x ′, y ′) relative to the reference coordinates G0 (x0, y0) as shown in the enlarged view of the pixels in FIG. The distance d (G) is calculated as an index for evaluating how close it is. Then, the pixel insertion processing unit 404 determines whether or not to perform pixel insertion using the value of the distance d (G). The pixel insertion processing unit 404 calculates the distance d (G) by the following equation (7). After step S803, the pixel insertion processing unit 404 advances the process to step S805.

  On the other hand, when the process proceeds from step S802 to step S804, the pixel insertion processing unit 404 determines that the pixel value Gpix is a missing pixel, and therefore, as the exception process, the maximum that can be expressed by the distance d (G). Enter a value. For example, when the distance d (G) can be expressed only up to 8 bits, “255” is input as the value of the distance d (G). After step S804, the pixel insertion processing unit 404 advances the process to step S805.

  In step S805, the pixel insertion processing unit 404 determines whether or not the variable M is greater than “1”. That is, in step S805, it is determined whether the M-th image currently being processed is the second or later non-reference image or the first reference image. If the pixel insertion processing unit 404 determines in step S805 that the variable M is greater than “1” and is processing of the second and subsequent non-reference images, the pixel insertion processing unit 404 proceeds to step S808. On the other hand, if the variable M is 1 in step S805 and the pixel insertion processing unit 404 determines that the process is for the first reference image, the process proceeds to step S806.

  When the processing proceeds to step S806, the pixel insertion processing unit 404 uses the distance d (G) obtained in step S803 as the distance between the coordinate G (x ′, y ′) and the reference coordinate G0 (x0, y0) in the standard image. It is stored in the memory unit 204 as d0 (G). Thereafter, the pixel insertion processing unit 404 uses the distance d0 (G) for comparison with the value of the distance d (G) obtained from the second and subsequent non-reference images. Then, the pixel insertion processing unit 404 updates the value of the distance d0 (G) as needed according to the value of the distance d (G). After step S806, the pixel insertion processing unit 404 advances the process to step S807.

  In step S807, the pixel insertion processing unit 404 stores the pixel value Gpix of the coordinate G (x ′, y ′) in the standard image in the memory unit 204 as the pixel value Gpix0 of the reference coordinate G0 (x0, y0). Then, the pixel insertion processing unit 404 newly creates a high-resolution image 1100 using only each pixel value Gpix0 for only the first reference image. The high-resolution image 1100 newly created using only the reference image using the pixel value Gpix0 is updated as needed by inserting pixel values from the second and subsequent images in the subsequent step S812. Become. After step S807, the pixel insertion processing unit 404 ends the pixel insertion process for the coordinates (x ′, y ′) currently being processed. Thereafter, the pixel insertion processing unit 404 starts the processing of the flowchart of FIG. 13 when processing the next image.

  When the process proceeds from step S805 to step S808, the pixel insertion processing unit 404 determines whether the distance d (G) is less than a predetermined threshold TH (G) set in advance for comparison with the distance. To do. If the pixel insertion processing unit 404 determines in step S808 that the distance d (G) is less than the threshold value TH (G), the process proceeds to step S809. On the other hand, if the pixel insertion processing unit 404 determines in step S808 that the distance d (G) is greater than or equal to the threshold value TH (G), the coordinates (x ′, y ′) currently targeted for processing are determined. ) Ends the pixel insertion process. Thereafter, the pixel insertion processing unit 404 starts the processing of the flowchart of FIG. 13 when processing the next image.

  When the process proceeds to step S809, the pixel insertion processing unit 404 reads the value of the distance d0 (G) stored for the reference coordinate G0 (x0, y0) from the memory unit 204. After step S809, the pixel insertion processing unit 404 advances the process to step S810.

  In step S810, the pixel insertion processing unit 404 compares the distance d (G) with the distance d0 (G). If the pixel insertion processing unit 404 determines in step S810 that the distance d (G) is smaller than the distance d0 (G), the pixel insertion processing unit 404 proceeds to step S811. On the other hand, if the pixel insertion processing unit 404 determines in step S810 that the distance d (G) is greater than or equal to the distance d0 (G), the coordinates (x ′, y ′) currently being processed are determined. This completes the pixel insertion process.

  In step S811, the pixel insertion processing unit 404 updates the value of the distance d0 (G) to the distance d (G) and stores it in the memory unit 204. After step S811, the pixel insertion processing unit 404 advances the process to step S812.

  In step S812, the pixel insertion processing unit 404 updates the pixel value Gpix corresponding to the distance d (G) as the pixel value Gpix0 and stores it in the memory unit 204. After storage in the memory unit 204, the pixel insertion processing unit 404 ends the pixel insertion processing of the coordinates (x ′, y ′) that are currently processed. The above is the detailed processing of step S708 performed by the pixel insertion processing unit 404.

Returning to FIG.
After step S708, the super-resolution processing unit 205 advances the process to step S709. In step S709, after the pixel insertion processing in step S708, the super-resolution processing unit 205 performs the processing from step S702 to step S708 on all the coordinates of the Mth image currently being processed. Determine if it is over. If the super-resolution processing unit 205 determines in step S709 that the processing for all coordinates has not been completed, the process proceeds to step S710. If the super-resolution processing unit 205 determines that the processing for all coordinates has been completed, the process proceeds to step S711. Proceed with the process.

  When the processing proceeds to step S710, the super-resolution processing unit 205 determines the coordinates R (x, y), G (x, y), and B (x, y) that are currently processed, The process is updated to the next coordinate, and the processes after step S702 are performed.

  When the processing proceeds to step S711, the super-resolution processing unit 205 determines whether or not the image currently being processed is the last N-th image. If the super-resolution processing unit 205 determines in step S711 that the image is the last N-th image, the super-resolution processing unit 205 proceeds to step S712. If the super-resolution processing unit 205 determines that the image is not the last N-th image, The process proceeds to step S714.

  The processing in step S712 is processing performed by the missing interpolation processing unit 405. In step S712, the missing interpolation processing unit 405 performs missing interpolation processing on the N-th high-resolution image for each color component of R, G, and B after the pixel insertion processing in step S708 is completed. . The missing interpolation processing performed in step S712 is performed by replacing pixels that are still missing pixels and have a value of zero in the high-resolution images for each of the R, G, and B color components from the surrounding non-missing pixels. This is a process of replacing the pixel value generated by the adaptive interpolation process with the direction taken into account. The high-resolution image for each color component of R, G, B after the missing interpolation processing in step S712 is an image in which all pixels are composed of non-missing pixels.

In this embodiment, adaptive interpolation processing is used as an example of missing interpolation processing. The missing interpolation process may be a linear interpolation process. Since these adaptive interpolation processing and linear interpolation processing are known techniques, the description thereof is omitted.
Further, the super-resolution processing unit 205 calculates the ratio of non-missing pixels in the image before the missing interpolation processing by the missing interpolation processing unit 405 is performed, and controls each processing unit according to the ratio. May be.

Hereinafter, a control example according to the ratio of non-missing pixels will be described with reference to FIG.
FIG. 15 is a diagram showing a high-resolution image 1200 in which the number of pixels in both the horizontal and vertical directions is increased by k times with respect to a low-resolution image having W and H pixel numbers in the horizontal and vertical directions. Here, for example, when the number of non-missing pixels 1202 is not 50% or more with respect to the total number of pixels of the high-resolution image 1200, the missing pixels 1201 are accurately generated during adaptive interpolation with the edge direction taken into account. Can not be. In this case, the finally obtained super-resolution image cannot be said to have high image quality, and the effect of the super-resolution processing is reduced. For this reason, the super-resolution processing unit 205 calculates the ratio of non-missing pixels in the super-resolution image by the following equation (8). Note that k in Expression (8) is an enlargement ratio when a super-resolution image is generated by enlarging a low-resolution image.

Ratio (%) = {(Total number of non-missing pixels in high-resolution image) / k2 × W × H} × 100 (%)
... Formula (8)

  The super-resolution processing unit 205 performs the above-described ratio calculation for the three high-resolution images for each of the R, G, and B color components, and any one of the three ratios for each of the color components. If it is 50% or more, the super-resolution processing is stopped as a super-resolution processing error. When a super-resolution processing error occurs, the super-resolution processing unit 205 displays a low-resolution image for each of the R, G, and B color components in the first image (reference image) in both the horizontal and vertical directions. A high-resolution image enlarged by k times may be output to the YUV conversion unit 206 at the subsequent stage.

  In addition, when any one of the above-described three ratios for each of the R, G, and B color components is 50% or more, the super-resolution processing unit 205 performs the processing up to step S701 in FIG. For example, a process of generating a high resolution image of k / 2 times may be performed.

After step S712, the super-resolution processing unit 205 shifts the process to step S713 performed by the direction-specific filtering processing unit 406.
In step S713, the direction-specific filtering processing unit 406 performs direction-specific filtering processing on the high-resolution image for each color component of R, G, B after the missing interpolation processing by the missing interpolation processing unit 405. The direction-specific filtering process in step S713 will be described later.

  After step S713, the super-resolution processing unit 205 advances the processing to step S714, stores the super-resolution image for each color component of R, G, and B after the direction-specific filtering processing in the memory unit 204 and stores the super-resolution image. The super-resolution process of the flowchart of FIG.

Hereinafter, the direction-specific filtering processing performed by the direction-specific filtering processing unit 406 in step S713 of FIG. 10 will be described in detail.
The direction-specific filtering process of the present embodiment is a process for performing filtering along the edge direction on a high-resolution image for each color component of R, G, and B. The direction-specific filtering processing unit 406 performs filtering processing along the edge direction, thereby reducing image quality degradation due to unnatural signal switching caused by a positional shift at the time of pixel insertion and reducing the image bandwidth as much as possible. I try not to let you. Further, the direction-specific filtering processing unit 406 also performs processing for suppressing an increase in noise when, for example, noise or the like is generated in a portion other than the edge.

FIG. 16 shows a flowchart of processing by the direction-specific filtering processing unit 406.
Hereinafter, the direction-specific filtering process will be described with reference to the flowchart of FIG. In FIG. 16, the filtering process for each direction with respect to the high resolution of the G component is described as an example. However, the same processing is applied to the R and B components. In FIG. , B component processing can be realized. For this reason, the description of the direction-specific filtering process for the high-resolution image of the R and B components is omitted.
The processing by the direction-specific filtering processing unit 406 is roughly performed by first determining the edge direction and then performing the filtering process along the edge direction based on the direction determination result.

  First, in step S1301, the direction-specific filtering processing unit 406 performs the initial tap number for the filter used for the filter processing for the edge direction determination processing and the filter used for the subsequent direction-specific filtering processing. Determine the number of initial taps. The number of taps is the number of directions when the direction determination is performed with respect to the pixels related to the pixel insertion process (hereinafter referred to as the pixel of interest) with the number of pixels around the pixel and the subsequent filtering process according to direction. Initially set as a number corresponding to the number of directions.

  In the case of the present embodiment, the number of initial taps used for the filter processing for the direction discrimination processing and the number of initial taps for the direction-specific filtering processing are specifically the following first and second parameters: Determined based on The first parameter is information indicating how many times the low-resolution image input to the image processing unit 104 is to be enlarged in the horizontal and vertical directions to generate a super-resolution image. The second parameter is information indicating how many times the generated super-resolution image is further enlarged in the horizontal and vertical directions to be an output image.

  FIG. 17 shows the initial number of taps of the direction determination process and the direction-specific filtering process, the enlargement ratio of the super-resolution image for the low-resolution image input to the image processing unit 104, and the output image for the generated super-resolution image. The relationship with the enlargement ratio is shown. As an example, when a super-resolution image is generated by enlarging the input low-resolution image twice in the horizontal and vertical directions, and the super-resolution image is used as an output image without being enlarged, it is used for direction discrimination processing. The number of initial taps to be performed and the number of initial taps for direction-specific filtering are 5 taps. That is, according to the present embodiment, the number of taps is changed according to the enlargement ratio of the super-resolution image with respect to the input low-resolution image and the enlargement ratio of the output image with respect to the super-resolution image. The number of directions when discriminating is determined.

Returning to FIG.
After the initial tap number setting processing in step S1301, the direction-specific filtering processing unit 406 advances the processing to step S1302.
In step S1302, the direction-specific filtering processing unit 406 initializes the coordinates G (x ′, y ′) of the G component high-resolution image to the coordinates G (0, 0). After step S1302, the direction-specific filtering processing unit 406 advances the process to step S1303.
Note that the processes from step S1303 to step S1312 are all performed for each coordinate.

  In step S1303, the direction-specific filtering processing unit 406 calculates the distance d0 (G) obtained from the coordinates G (x ′, y ′) of the G component high-resolution image and the reference coordinates G0 (x0, y0) described above. Obtained as the distance d0 (G) of the inserted pixel to be inserted. The distance d0 (G) at this time is acquired as a value representing the positional deviation amount of the non-reference image with respect to the first reference image. Therefore, the distance d0 (G) is an index indicating how much the second and subsequent non-reference images are deviated from the coordinates of the first reference image. After step S1303, the direction-specific filtering processing unit 406 proceeds to step S1304.

  In step S1304, the direction-specific filtering processing unit 406 determines the direction of the high-resolution image of the G component according to the distance d0 (G) of the inserted pixel corresponding to the coordinate G (x ′, y ′) of the target pixel. It is determined whether or not processing can be performed. This determination can be made, for example, by comparing the value of the distance d0 (G) with a predetermined threshold value TH (G) as in the comparison process in step S808 of FIG. For example, the direction-specific filtering processing unit 406 determines that the direction determination processing can be performed on the high-resolution image of the G component when the distance d (G) is less than a predetermined threshold TH (G). Further, whether or not the direction determination process can be performed is determined by a predetermined value set in advance for comparison between the ratio of the non-missing pixels to the high-resolution image of the G component calculated by the above-described equation (8) and the ratio. It can also be performed by comparison with a threshold value (for example, 50%). For example, the direction-specific filtering processing unit 406 determines that the direction determination processing can be performed on the high-resolution image of the G component when the ratio of the non-missing pixels is less than the threshold (50%). If the direction-specific filtering processing unit 406 determines in step S1304 that the direction determination processing can be performed on the high-resolution image of the G component, the process proceeds to step S1305. On the other hand, if the direction-specific filtering processing unit 406 determines in step S1304 that the direction-specific determination process cannot be performed, the direction-specific filtering process is performed using the direction determination result of the low-resolution image, and the process is performed in step S1307. To proceed.

  In step S1305, the direction-specific filtering processing unit 406 determines the number of direction determination taps used in the direction determination processing of the coordinates G (x ′, y ′) of the pixel of interest. The number of taps at this time is determined by determining whether or not to increase from the initial number of taps determined in step S1301 described above. The determination in this case is made by comparing the value of the distance d0 (G) of the inserted pixel corresponding to the coordinate G (x ′, y ′) of the pixel of interest with a predetermined threshold set in advance for comparison with the distance. Or may be determined according to the coordinates and pixel values at that time. In the case of comparison with the threshold, as an example, when the value of the distance d0 (G) of the inserted pixel corresponding to the coordinate G (x ′, y ′) of the target pixel is less than the threshold, the direction-specific filtering processing unit 406 The number of taps used in the direction discrimination process is increased from the initial number of taps. After step S1305, the direction-specific filtering processing unit 406 proceeds to step S1306.

  In step S1306, the direction-specific filtering processing unit 406 performs direction determination processing on the coordinates G (x ′, y ′) of the target pixel of the G component high-resolution image using the surrounding pixels. The direction determination processing in this case is the same as the direction determination processing by the above-described direction determination processing unit 304. Here, a case where the direction determination processing is performed with five taps will be described.

A direction determination process using five taps will be described with reference to FIGS.
FIG. 18A to FIG. 18L illustrate a target pixel 1501 and peripheral pixels 1502 to 1525 used for direction determination, respectively. In the case of the present embodiment, the direction-specific filtering processing unit 406, as shown in FIGS. 18 (a) to 18 (l), for the target pixel 1501, the target for direction determination among the surrounding pixels 1502-1525. Using the values of the four target pixels, a total of 12 directions are discriminated.

  Since description of all the 12 directions shown in FIGS. 18A to 18L will be complicated, only FIG. 18A will be given as an example. In the example of FIG. 18A, the pixel value of the target pixel 1501 is set to pix, and the pixel values of the direction determination target pixels 1502, 1503, 1504, and 1505 are set to pix1, pix2, pix3, and pix4, respectively. The direction-specific filtering processing unit 406 calculates the sum of absolute differences between the pixel value pix of the target pixel 1501 and the pixel values pix1, pix2, pix3, and pix4 of the target pixels 1502, 1503, 1504, and 1505 by the following equation (9). Calculate sad. In the equation (9), pixi is the pixel values pix1 to pix4 of the target pixels 1502 to 1505.

  The direction-specific filtering processing unit 406 obtains a difference absolute value sum sad for each of FIGS. 18B to 18L as in the example of FIG. The difference absolute value sum sad obtained for each of the 12 directions in FIGS. 18A to 18L is defined as sad1 to sad12. The direction-specific filtering processing unit 406 obtains the difference absolute value sum min_sad having the smallest difference absolute value sum among the difference absolute value sums sad1 to sad12 in the 12 directions. Then, the direction-specific filtering processing unit 406 sets the minimum difference absolute value sum min_sad as the direction obtained by the direction determination process.

  However, the direction-specific filtering processing unit 406 does not correspond to all the 12 directions when the difference between the minimum difference absolute value sum min_sad and each of the other difference absolute value sums is equal to or less than a predetermined threshold value set in advance. And outputs an error as the direction discrimination result. Note that the threshold for the difference between the minimum difference absolute value sum min_sad and each of the other difference absolute value sums may be changed for each scene of the image as an arbitrary parameter.

  Note that, in the present embodiment, the processing described above is performed using the high-resolution image of the G component that is the target of the direction-specific filtering processing for the processing of step S1305 and step S1306. Processing using a resolution image may be used. After step S1306, the direction-specific filtering processing unit 406 proceeds to step S1310.

  Here, the processing in steps S1305 and S1306 is direction determination processing using a high-resolution image. On the other hand, in the following steps S1307 to S1309, direction discrimination processing using the direction discrimination results of a plurality of input images is performed. In other words, when it is determined in step S1304 that the direction determination process cannot be performed, in steps 1307 to S1309, the direction filtering process is performed using the direction determination result of the low resolution image.

  When the processing proceeds to step S1307, the direction-specific filtering processing unit 406 acquires the direction discrimination result stored in the memory unit 204 by the above-described direction discrimination processing unit 304. At this time, the direction-specific filtering processing unit 406 acquires the direction determination result corresponding to the coordinates G (x ′, y ′) for the plurality of low-resolution images. Note that the coordinates (x, y) of the low-resolution image corresponding to the coordinates G (x ′, y ′) are coordinates (x ′) obtained by dividing the coordinates (x ′, y ′) by k, which is the magnification described above. / K, y ′ / k). After step S1307, the direction-specific filtering processing unit 406 proceeds to step S1308.

  In step S1308, the direction-specific filtering processing unit 406 performs a process of totaling the direction determination results of the low-resolution images corresponding to the number of input images acquired in step S1307 (totaling the number of determination results for the same direction). .

In the following, a process of aggregating the direction determination results of the low resolution images corresponding to the number of input images will be described with reference to FIGS. 19 (a) to 19 (e) and 19 (f).
In addition, the example of FIG. 19A-FIG. 19D has shown the result of having discriminate | determined the direction discrimination | determination result of the low-resolution image by 3 taps. FIGS. 19A to 19D show a pixel of interest 1701 and peripheral pixels 1702 to 1709 used for direction determination, respectively.

  The direction-specific filtering processing unit 406 aggregates the discrimination results in four directions using the values of the target pixels, each of which is a target for direction discrimination among the surrounding pixels 1702 to 1709, with respect to the target pixel 1701. . FIG. 19A shows the target pixels 1702 and 1703, FIG. 19B shows the target pixels 1704 and 1705, FIG. 19C shows the target pixels 1706 and 1707, and FIG. 19D shows the target pixel 1701. In this example, the directions of 1708 and 1709 are determined. Note that FIG. 19E shows a direction determination result when the direction cannot be determined in all four directions (when an error is determined). Here, the direction discrimination results shown in FIGS. 19A to 19E are defined as detect [0], detect [1], detect [2], detect [3], detect [err]. The direction-specific filtering processing unit 406 performs each direction determination result on each of detect [0], detect [1], detect [2], detect [3], detect [err] as shown in FIG. The direction determination results of the low resolution images are summed for the number of input sheets.

Returning to FIG.
After step S1308, the direction-specific filtering processing unit 406 proceeds to step S1309.
In step S1309, the direction-specific filtering processing unit 406 acquires, as the direction determination result corresponding to the coordinate G (x ′, y ′) of the pixel of interest, the direction having the largest total number of the totals in step S1308. After step S1309, the direction-specific filtering processing unit 406 proceeds to step S1310.

  In step S1310, the direction-specific filtering processing unit 406 determines the number of taps when performing the direction-specific filtering processing. The number of taps at this time is determined depending on whether the direction determination result is acquired by the processing from step S1305 to step S1306 or the direction determination result is acquired by the processing from step S1307 to step S1309. When the direction discrimination result is acquired by the processing from step S1305 to step S1306, the direction-specific filtering processing unit 406 uses the same tap number as the direction discrimination tap determined in step S1305. On the other hand, the direction-specific filtering processing unit 406 uses, for example, 3 taps when the direction determination result is acquired by the processing from step S1307 to step S1309. Although detailed description will be given later, if the direction determination result is an error output, the direction-specific filtering processing unit 406 uses 3 × 3 taps. After step S1310, the direction-specific filtering processing unit 406 proceeds to step S1311.

  In step S1311, the direction-specific filtering processing unit 406 determines a filter coefficient for performing the direction-specific filtering processing according to the number of taps determined in step S1310.

  Hereinafter, filter coefficient determination processing for performing direction-specific filtering processing will be described with reference to FIGS. 20 (a) to 20 (f). Here, an example in which the number of taps is 5 taps is given. Further, in the present embodiment, as an example of the direction determination result, a determination result by the target pixel 1801 and the target pixels 1802 to 1805 is obtained as illustrated in FIG.

  When determining the filter coefficients of the pixel of interest 1801 and the target pixels 1802 to 1805 based on the result of direction discrimination as in the example of FIG. 20A, the direction-specific filtering processing unit 406 first, as shown in FIG. A temporary filter coefficient is determined. For example, as a 5-tap filter coefficient, the filter coefficient of the target pixel 1801 is “6”, the filter coefficients of the target pixels 1802 and 1805 are “1”, and the filter coefficients of the target pixels 1803 and 1804 are “4” (1, Filter coefficients of 4, 6, 4, 1) are used. In the case of a 3-tap filter coefficient, for example, the filter coefficient (1, 2, 1) in which the filter coefficient of the pixel of interest is “2” and the filter coefficients of the two target pixels is “1” is used. .

  As shown in FIG. 20C, the direction-specific filtering processing unit 406 acquires from the memory unit 204 the distance d0 (G) at the time of pixel insertion in the target pixels 1802 to 1805 and the target pixel 1801. Here, the distance in the target pixel 1801 is d0 (0), and the distances in the target pixels 1802, 1803, 1804, and 1805 are d0 (−2), d0 (−1), d0 (1), and d0 ( 2). In the case of this example, the direction-specific filtering processing unit 406 creates a weight table as shown in FIG. 20D based on the distance d0 (0) acquired in the example of FIG. The weight table is a weight table in which the weight of the distance d0 (0) is “1” and the maximum weight is “2”, and is created with a predetermined threshold value based on the distance d0 (0). Further, the direction-specific filtering processing unit 406 uses the weight table to determine the weight w (0) of the target pixel 1801, the weight w (-2) of the target pixel 1802, the weight w (-1) of the target pixel 1803, and the target pixel 1804. The weight w (1) and the weight w (2) of the target pixel 1805 are calculated. Then, the direction-specific filtering processing unit 406 multiplies the temporary filter coefficient in FIG. 20B by the weight in FIG. 20D to determine a filter coefficient as shown in FIG.

If the direction discrimination result is an error output, the direction-specific filtering processing unit 406 uses each predetermined 3 × 3 tap filter coefficient as shown in FIG.
In the present embodiment, the example of determining the filter coefficient for performing the direction-specific filtering process by the method described in FIGS. 20A to 20F has been described. However, the present invention is not limited to this method. For example, FIG. A predetermined filter coefficient such as the temporary filter coefficient of b) may be used.

Returning to FIG. After step S1311, the direction-specific filtering processing unit 406 advances the process to step S1312.
In step S1312, the direction-specific filtering processing unit 406 performs direction-specific filtering using the filter coefficient determined in step S1311. Here, for example, the pixel value of the pixel of interest is pix0, and the pixel values of the target pixel whose direction is determined in four directions are pix1, pix2, pix3, and pix4, respectively. Further, if the filtering coefficients of each pixel are f [0], f [1], f [2], f [3], and f [4], the direction-specific filtering process for 5 taps is as follows: This is a process of calculating the pixel value pix of the target pixel after filtering using (10).

  In addition, although 5 taps were mentioned here as an example, also in the case of 3 taps, the filtering process according to a direction and the filtering process when a direction discrimination | determination result becomes an error output are the same. After step S1312, the direction-specific filtering processing unit 406 proceeds to step S1313.

  In step S1313, the direction-specific filtering processing unit 406 determines whether or not the processing in step S1312 described above has been completed for all the pixels of the G component high-resolution image. The direction-specific filtering processing unit 406 proceeds to step S1315 if it is determined in step S1313 that the process has been completed, and proceeds to step S1314 if it is determined that the process has not been completed.

  When the processing proceeds to step S1314, the direction-specific filtering processing unit 406 updates the coordinates G (x ′, y ′) of the G component high-resolution image to the coordinates of the next pixel, and returns the processing to step S1303.

On the other hand, in step S1315, the direction-specific filtering processing unit 406 displays the image after the direction-specific filtering processing is performed on all the pixels of the G component high-resolution image in the processing up to step S1314 described above as the G component. Is output as the final super-resolution image. Thereafter, the direction-specific filtering processing unit 406 ends the direction-specific filtering processing of FIG.
The direction-specific filtering processing unit 406 performs the above-described processing on the R component high-resolution image and the B component high-resolution image in the same manner, so that final R and B component super-resolution images are obtained. Output as.

  As described above, according to the present embodiment, by performing the filtering process along the edge direction, an unnatural signal switching occurs due to the positional deviation of the pixels without reducing the band of the image as much as possible. It is possible to reduce image quality degradation due to the above. Further, according to the present embodiment, even when noise or the like is generated in a portion other than the edge, an increase in the noise can be suppressed.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  DESCRIPTION OF SYMBOLS 101 Optical system, 102 Imaging device, 103 A / D conversion part, 104 Image processing part, 105 System control part, 106 Operation part, 107 Display part, 108 Recording part, 109 Bus, 201 Reference | standard image selection part, 202 Bayer separation part , 203 coordinate conversion coefficient calculation unit, 204 memory unit, 205 super-resolution processing unit, 206 YUV conversion unit, 301 demosaic processing unit, 302 distortion correction processing unit, 303 projective conversion coefficient calculation unit, 401 magnification chromatic aberration coordinate correction unit, 402 Distorted coordinate correction unit, 403 coordinate conversion unit for alignment, 404 pixel insertion processing unit, 405 missing interpolation processing unit, 406 filtering by direction

Claims (15)

An image processing apparatus that generates a high-resolution image using a plurality of low-resolution images,
Positioning means for aligning positions of a plurality of other images excluding the reference image with respect to the reference image, using one image as a reference for alignment among the plurality of input images as a reference image When,
Pixel insertion means for generating a high-resolution image by inserting pixels of the other plurality of images with respect to the reference image;
The direction of the edge is determined from the image after the pixel is inserted, and filtering according to the direction of the edge is performed on the image after the pixel is inserted based on the determined direction of the edge. And an image processing apparatus.   The filter means performs a filter process for determining an edge direction using a pixel value of a target pixel in an image after the pixel is inserted and pixel values of a plurality of pixels around the target pixel. The image processing apparatus according to claim 1, wherein the image processing apparatus determines an edge direction of the image after the pixel is inserted. The filter means includes
Based on the coordinate value set for the reference image to perform the alignment, the positional deviation amount of the other plurality of images with respect to the reference image is obtained,
Based on the positional deviation amount, it is determined whether or not the direction of the edge can be determined from the image after the pixel is inserted, and the filtering according to the direction is performed based on the direction of the edge that has been determined. The image processing apparatus according to claim 1, wherein the image processing apparatus performs the processing.   The said filter means determines that the direction of an edge can be discriminate | determined from the image after the said pixel is inserted, when the said amount of position shift is less than a predetermined threshold value. Image processing device.   The filter means determines the number of taps used in filter processing when determining the direction of the edge from the image after the pixel is inserted based on the positional deviation amount. 5. The image processing apparatus according to 4.   The filter means determines a filter coefficient for each tap used in a filter process when determining an edge direction from an image after the pixel is inserted based on the positional deviation amount. The image processing apparatus according to any one of 3 to 5.   The said filter means determines the number of directions at the time of discriminating the direction of the said edge from the image after the said pixel is inserted based on the said amount of positional offsets. The image processing apparatus according to item 1.   The filter means determines the number of directions when determining the direction of the edge from the image after the pixel is inserted according to the resolution of the image after the pixel is inserted. Item 7. The image processing apparatus according to any one of Items 1 to 6. The filter means includes
For the image after the insertion of the pixel, find the ratio of the number of missing pixels whose pixel value is zero and the number of non-missing pixels whose pixel value is greater than zero,
Based on the ratio, it is determined whether or not the direction of the edge can be determined from the image after the pixel is inserted, and the filtering according to the direction is performed based on the direction of the edge that has been determined. The image processing apparatus according to claim 1, wherein:   The filter means obtains a ratio of non-missing pixels in the image after the insertion of the pixels as a ratio between the number of missing pixels and the number of non-missing pixels, and the ratio is a predetermined threshold value. The image processing apparatus according to claim 9, wherein if it is less than the threshold value, it is determined that an edge direction can be determined from an image after the pixel is inserted. Performing a filtering process for determining an edge direction using a pixel value of a pixel of interest in an image before the pixel is inserted and pixel values of a plurality of pixels around the pixel of interest; Having direction discriminating means for discriminating the direction of the edge of the image before being inserted,
If it is determined that the edge direction cannot be determined from the image after the pixel is inserted, the filter unit inserts the pixel based on the edge direction determined by the direction determination unit. The image processing apparatus according to claim 3, wherein filtering is performed for each edge direction on the image after being processed. The direction discriminating unit discriminates the direction of the edge using the reference image before the pixel is inserted and the other plural images, respectively.
When it is determined that the direction of the edge cannot be determined from the image after the pixel is inserted, the filter means determines whether the reference image before the pixel is inserted and the other plurality of images. The directions of a plurality of determined edges are totaled for each direction, and filtering according to the direction of the edge is performed on the image after the pixels are inserted based on the direction having the largest total number. The image processing apparatus according to claim 11. The filter means includes
When the edge direction can be determined from the image after the pixel is inserted, the number of taps used in the filter processing when determining the edge direction from the image after the pixel is inserted is Determine the number of taps when filtering by edge direction,
When it is determined that the direction of the edge cannot be determined from the image after the pixel is inserted, the number of taps used in the filtering process when the direction determining unit determines the direction of the edge is The image processing apparatus according to claim 11, wherein the number of taps is determined as the number of taps for filtering in each direction. An image processing method for generating a high-resolution image using a plurality of low-resolution images,
The alignment means uses, as a reference image, one image serving as a reference for alignment among the plurality of input images, and positions of a plurality of other images excluding the reference image with respect to the reference image The step of matching
A step of inserting a pixel of the plurality of other images with respect to the reference image to generate a high-resolution image;
Filter means discriminates the direction of the edge from the image after the pixel is inserted, and based on the discriminated edge direction, the image after the pixel is inserted is classified according to the direction of the edge. An image processing method comprising the steps of:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-13.

Images