JP6579934B2 - Image processing apparatus, imaging apparatus, image processing method, program, and storage medium - Google Patents

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that corrects moire in a captured image.

  Patent Document 1 uses a multi-lens imaging device to shoot a plurality of images so that the frequency of moire generated in the same portion of the subject image is different, and analyze the frequency components of the plurality of images. Thus, a method for detecting and correcting moire has been proposed.

JP 2003-18479 A

  However, in the imaging apparatus described in Patent Document 1, by controlling the frequency of moire generated in the same part of the subject image to be different, a part of the subject area where moire is to be detected is detected in an image under a specific shooting condition. In some cases, the angle of view may be outside the field of view.

  For example, suppose that the focal length of the optical system is changed, and two types of images of wide angle and telephoto are imaged with an imaging element having the same pixel pitch. In this case, even if there is a moiré subject in a region away from the center of the optical axis in the wide-angle image, only a partial region corresponding to the subject can be captured in the telephoto image. There is. As described above, when there is no telephoto subject area that has a one-to-one correspondence with the wide-angle side, moire cannot be detected and corrected in that area, and there is a problem that image quality deteriorates.

  In view of the above problems, an object of the present invention is to provide an image processing apparatus that is advantageous for moire correction.

  The present invention is characterized in that, for a non-common area different from the common area among a plurality of images including the common area, the similarity with the moire area detected in the common area is calculated. Then, based on the detected moire area and the calculated similarity, the moire of a plurality of images is corrected.

  According to the present invention, it is possible to provide an image processing apparatus that is advantageous for moire correction.

It is a block diagram which shows the structure of the imaging device concerning the Example of this invention. It is a flowchart which shows the flow of the image generation process concerning the Example of this invention. It is a flowchart which shows the flow of a process of the similarity calculation means concerning the Example of this invention. It is a figure explaining the effect | action of a moire detection and reduction process in Example 1. FIG. It is a figure explaining the effect | action of a moire detection and reduction process in Example 2. FIG. It is a schematic diagram which shows the pixel arrangement | sequence of the image pick-up element concerning the Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating a schematic configuration of an imaging unit 100 included in the imaging apparatus 10 of the present invention. The operation of the imaging unit 100 is controlled by the control unit 109. The control unit 109 performs overall control of the entire imaging apparatus 10.

  The imaging device (imaging unit) samples a continuous subject image formed by a photographic lens (optical system) with an imaging element such as a CMOS sensor, converts it to digital data, performs digital signal processing, A video signal including color information is generated. Here, when a continuous subject image is sampled by the image sensor, if the subject image has a frequency component equal to or higher than the Nyquist frequency determined by the pixel pitch of the image sensor, aliasing distortion (moire) occurs and the generated video signal Image quality may be impaired. Note that the imaging apparatus of the present invention uses a multi-lens imaging apparatus including a plurality of optical systems and imaging elements. In a multi-lens imaging device, a plurality of optical systems and a plurality of imaging elements are controlled independently, and the same subject can be imaged under different optical conditions and imaging sampling conditions. In the present embodiment, the imaging apparatus of the present invention is exemplified by a lens-integrated imaging apparatus. However, the present invention is not limited to this, and the present invention is not limited to this. Can also be applied.

  Hereinafter, the components provided in the imaging unit 100 will be described with reference to FIG.

  The imaging unit 100 includes a first imaging unit 101, a second imaging unit 102, and an image processing unit 110 (image processing apparatus). The first imaging unit 101 and the second imaging unit 102 each include an optical system, an imaging device, and a simultaneous interpolation unit. Here, the first imaging unit 101 and the second imaging unit 102 can independently control the optical conditions of the optical system and the driving conditions of the imaging element, and the same subject can be controlled under different imaging conditions. You can take an image.

  In this embodiment, imaging is performed by changing the focal lengths of the optical systems in the first imaging unit 101 and the second imaging unit 102 so that the frequency of moire generated in the same portion of the subject image is different.

  The simultaneous interpolation means interpolates the output signal of the image sensor and generates luminance and color signals.

  Note that the image pickup apparatus of the present embodiment is configured to have two image pickup means, but may have three or more independently controllable image pickup means.

  The image processing unit 110 includes acquisition units 110a and 110b that acquire a plurality of images (signals) output from the first imaging unit 101 and the second imaging unit. The plurality of images includes a common area for enabling determination of a moire area as will be described later. The plurality of images acquired by the acquisition units 110a and 100b are sent to the geometric deformation unit 103, the similarity calculation units 105 and 106, and the moire correction units 107 and 108.

  The geometric deformation unit 103 performs geometric deformation on the images output from the first imaging unit 101 and the second imaging unit 102. More specifically, geometrical image quality differences caused by optical characteristics or imaging sampling characteristics are corrected for a plurality of images output from the imaging means. In a multi-eye imaging device, parallax occurs depending on the in-focus state. Therefore, prior to comparing the frequency components corresponding to the same subject area among the plurality of images, the geometrical deformation means 103 corrects the deviation of the subject image due to the parallax.

  The captured images output from the first imaging unit 101 and the second imaging unit 102 have different image magnifications due to differences in focal length. Therefore, prior to comparing the frequency components corresponding to the same subject area between the plurality of images, the geometric deformation means 103 performs a scaling process so that the image magnifications are equal.

  The moiré area detection unit 104 refers to the output image of the geometric deformation unit 103, analyzes the color signal and the frequency component in the same subject area, and determines whether or not moire is generated due to the sampling of the image sensor. To do.

  Specifically, the moire area detection unit 104 is an area where the same subject image is captured from a plurality of images output from the geometric deformation unit 103 and captured under different conditions (hereinafter referred to as the same subject area). Is detected. Then, a color signal and a frequency component are calculated by calculating pixel values in the same subject area. Then, if there is a difference in feature amount between a plurality of images taken under different conditions in the same subject area, it is determined that moire has occurred. As described above, the moire area detection unit 104 functions as a determination unit that determines an area (moire area) in which moire occurs in the same subject area (common area) among a plurality of images. Further, the moire area detection unit 104 determines a common area in which the moire area can be detected (determined) and a non-common area in which the moire area cannot be detected (determined) based on the difference between the plurality of images.

  Similarity calculation means 105 and 106 calculate the similarity of the image with the moiré area detected by the moiré area detection means 104 in the output images of the first imaging means 101 and the second imaging means 102, respectively. Specifically, regarding a non-common area different from a common area among a plurality of images (that is, an image area different from an image area common to a plurality of images and not common to a plurality of images), a moire area and It functions as a calculation means for calculating the degree of similarity.

  The processing contents of the similarity calculation unit 105 and the similarity calculation unit 106 are the same. The similarity calculation unit 105 calculates the similarity between the output of the moire region detection unit 104 and the output image of the second imaging unit 102. The similarity calculation unit 106 calculates the similarity between the output of the moire region detection unit 104 and the output image of the first imaging unit 102.

  In the present embodiment, the same subject area is imaged only in one of the output image of the first imaging means and the output image of the second imaging means, and the other output image has the same Only a part of the subject area is imaged. The similarity calculation means 105, 106 is based on the feature amount of the image of the area where the moire area detection means 104 cannot detect the moire area and the neighboring area determined as moire by the moire area detection means 104. Calculate similarity.

  Details of the processing contents of the similarity calculation means 105 and 106 will be described later with reference to the flowchart of FIG.

  The moiré correction means 107 and 108 refer to the results of the moiré area detection means 104 and the similarity calculation means 105 and 106, respectively, for the output images of the first imaging means 101 and the second imaging means 102, respectively. Performs processing to reduce moiré. As described above, the moire correction means 107 and 108 apply at least one of the plurality of images based on the moire area detected by the moire area detection means 104 and the similarity calculated by the similarity calculation means 105 and 106. Function as correction means for correcting moire. Specifically, with respect to the output images of the first imaging unit 101 and the second imaging unit 102, the similarity between the moire area detection result and the moire area is referred to for each pixel, and the moire area is detected. In some cases, a process of attenuating the moire signal level is performed. In addition, referring to the similarity between the moire area detection result and the moire area, if the area has a high degree of similarity to the moire area, a process for attenuating the moire signal level is performed.

  As an example of the process of attenuating the signal level in the moire area, in the case of a color signal, there is a process of reducing the saturation in order to mitigate the false color caused by moire. Further, in the case of a luminance signal, smoothing processing using surrounding signals can be cited to reduce the backlash of edges due to moire. Further, the signal level of the moire area may be attenuated by synthesizing the output images of the first imaging means and the second imaging means.

  In addition, in a region that is not a moire region and has a low similarity to the moire region, the degree of moire correction processing is weakened as the region is not similar to the moire region. In other words, the more similar to the moire area, the stronger the degree of moire correction processing. That is, the moire correction means 107 and 108 change the degree of correction according to the similarity calculated by the similarity calculation means 105 and 106 in a non-common area of a plurality of images.

  Note that the image processing apparatus of the present invention is not limited to the one mounted on the imaging apparatus, and may be provided outside the imaging apparatus. For example, it may be configured from a server connected to the imaging device via the Internet (cloud computing). Of course, the image processing apparatus may be connected to another personal computer (PC) storing image information, a mobile phone, a dedicated terminal such as a PDA or a game machine via a network.

  Next, the flow of processing for generating an image for one frame by the imaging unit of the present embodiment will be described with reference to the flowchart of FIG.

  In S100, the control unit 109 controls the driving of the optical system and the image sensor so that the first imaging unit and the second imaging unit have different imaging characteristics. Then, after the shooting conditions of the first image pickup means and the second image pickup means are determined in S100, shooting is executed in S101. In this embodiment, the focal length of each optical system is set so that the first imaging unit 101 captures a wide-angle image and the second imaging unit 102 captures a telephoto image.

  In S <b> 102, the geometric deforming unit 103 performs image displacement due to parallax and a difference in focal length on a plurality of images (output images) captured by the first imaging unit 101 and the second imaging unit 102. A geometric deformation process is performed so as to correct the difference in magnification.

  In S <b> 103, the moire area detection unit 104 acquires a plurality of images that are captured by the first imaging unit 101 and the second imaging unit 102 and deformed by the geometric deformation unit 103. Then, with reference to the plurality of acquired images, it is determined whether or not moire due to sampling of the image sensor has occurred, based on a difference in color signal or frequency component in the same subject region.

  Further, in the moiré area detection unit 104, a pixel on which the moiré area detection processing has been performed on the output images of the first imaging unit 101 and the second imaging unit 102 based on the difference between the two images, respectively. Moire detection target area information indicating whether or not (area) is generated. In addition, in the output image of the first imaging unit 101 and the second imaging unit 102, in the pixel area where the moiré detection process has been performed, the reference number of the moiré area indicating the moiré area to which it belongs is generated for each pixel. . The moire detection target area information and the reference number of the moire area correspond to the pixel positions of the output images of the first image pickup means 101 and the second image pickup means 102, respectively, and are not shown in the control means 109. Stored in

  In S <b> 104, the similarity calculation units 105 and 106 are similar in image to the moire area detected by the moire area detection unit 104 for the output images of the first imaging unit 101 and the second imaging unit 102, respectively. Calculate the degree.

  In S <b> 105, the moire correction means 107 and 108 apply to the output images of the first image pickup means 101 and the second image pickup means 102 based on the detection result of the moire area and the calculated similarity to the moire area. And correct the moire respectively. Specifically, a moire correction process for attenuating moire is performed.

  Next, the processing content of the similarity calculation means 105 and 106 will be described in detail with reference to the flowchart of FIG.

  In S201, the processing result of the moiré area detection unit 104 is referred to, and when the moiré area does not exist, the similarity calculation process ends. When the moiré area exists, the process proceeds to S202.

  In S202, the moire detection target area information is referred to from the storage means of the control means 109, and if there is an area in the input image that has not been subjected to moire detection processing by the moire area detection means 104, the process proceeds to S203. The similarity calculation process is continued. If there is no area where the moiré detection process has not been performed, the similarity calculation process ends.

  In S203, referring to the moiré detection target area information of the input image, the area where the moiré detection is not performed by the moiré area detection means 104 (non-common area) is set as the similarity calculation area. Then, for each pixel in the similarity calculation area, a feature amount used for similarity calculation is calculated and stored in a storage unit in the control unit 109.

  In S204, the moiré detection results obtained by the moiré area detection unit 104 are clustered, and the feature amount in the continuous moiré area is calculated. When there are a plurality of continuous moire areas, the feature amount is calculated for each moire area and stored in the storage means in the control means 109 together with the reference number of the moire area.

  Here, the feature amount calculated in S203 and S204 is information related to at least one of information such as a color signal, a dominant frequency component, and a subject distance. The information is generated based on at least the input image and the control setting of the imaging unit, and is used to estimate whether the pixel area close to the moire area is the same subject area as the moire area.

  In S205, the feature amount of the image obtained in S203 is referred to for the target pixel in the similarity calculation area.

  In S206, it is determined whether or not the difference in subject distance between the target pixel in the similarity calculation region (non-common region) and the target moire region is smaller than a predetermined threshold (first threshold).

  Specifically, Dh = (D (p) −D (mi)) ^ where D (p) is the subject distance at the pixel of interest P and D (mi) is the subject distance at the i th moire region of interest. It is determined whether 2 is smaller than a predetermined threshold value Thd (is less than or equal to the threshold value).

  Here, when the condition is satisfied, the target moire area and the target pixel are regarded as the same subject, and the signal of the target pixel is determined to be subjected to correction equivalent to that of the adjacent moire area. The process proceeds to S207. If the condition is not satisfied, the moiré area of interest and the pixel of interest are considered not to be the same subject, and the process proceeds to S212. In other words, the similarity calculation means 105 and 106 determine that the similarity is calculated when the difference in the subject distance is smaller than the first threshold, and when the difference is larger than the first threshold, the similarity is calculated. Is determined not to be calculated.

  In S207, it is determined whether or not the color signal difference between the target pixel in the similarity calculation region (non-common region) and the target moire region is smaller than a predetermined threshold (second threshold).

  Specifically, Ch = (C (p) −C (mi)) ^ where C (p) is the color signal at the pixel of interest P and C (mi) is the color signal at the i th moire region of interest. 2 is smaller than a predetermined threshold Thc (whether it is equal to or less than the threshold). Also, C (p) and C (mi) are expressed as linear sums of R, G, and B color components, respectively.

  Here, if the condition is satisfied, the target moire area and the target pixel are the same subject, and the similarity of the color signal is high, so the signal of the target pixel is corrected to be equal to the adjacent moire area. The process proceeds to S208.

  If the condition is not satisfied, the similarity of the color signals is low, so it is temporarily determined that the same correction as that of the adjacent moire area should not be performed on the target pixel, and the process proceeds to S209. In other words, the similarity calculation means 105 and 106 calculate the similarity when the difference between the color signals is smaller than the second threshold, and temporarily similar when the difference is larger than the second threshold. Do not calculate degrees.

  In S209, it is determined whether the difference between the frequency components of the pixel of interest in the similarity calculation region (non-common region) and the moire region of interest is smaller than a predetermined threshold (third threshold).

  Specifically, Fh = (F (p) −F (mi)) ^ where F (p) is the frequency component at the pixel of interest P and F (mi) is the frequency component at the i th moire region of interest. Whether 2 is smaller than a predetermined threshold value Thf (determined below a threshold value) is determined. F (p) and F (mi) are expressed as linear sums of signal components extracted by a band pass filter that maximizes the output in specific frequency bands in the horizontal, vertical, and diagonal directions, respectively. And

  Here, if the condition is satisfied, the target moire area and the target pixel are the same subject, and the similarity of the frequency components is high, so the signal of the target pixel is corrected to be equal to the adjacent moire area. The process proceeds to S208.

  If the condition is not satisfied, the similarity of the frequency components is low, so it is determined that the pixel of interest should not be corrected in the same way as the adjacent moire area, and the process proceeds to S212. In other words, the similarity calculation means 105 and 106 calculate the similarity when the frequency component difference is smaller than the third threshold, and calculate the similarity when the difference is larger than the third threshold. do not do.

  In S208, the reference number of the moire area adjacent to the target pixel and the similarity between the moire area and the target pixel are stored in a storage unit (not shown) in the control unit 109.

  As the similarity, the square value of the difference between the color signal or the frequency component between the moiré area and the pixel of interest calculated in S207 or S208 is stored.

  As described above, the similarity calculation units 105 and 106 calculate the similarity based on the feature amount of the pixel of interest in the non-common region and the feature amount of the moire region in the common region among the plurality of images. Specifically, the square value of the color signal difference or the frequency component difference is calculated as the similarity from the feature amount, and the calculated similarity is stored in a storage unit (not shown) in association with the focused moire area. .

  In S210, it is determined whether or not the calculation processing of the similarity with the moiré area of interest has been completed for all the pixels in the similarity calculation area.

  The address of the target pixel area is updated in S211 and the processes from S205 to S211 are repeated until the calculation processing of the similarity with the target moire area is completed for all the pixels. On the other hand, when the processing is completed for all the pixels, the process proceeds to S212.

  In S <b> 212, it is determined whether the similarity with the target pixel in the similarity calculation area has been evaluated for all target moire areas.

  For all moire areas, the reference number of the moire area is updated in S213 until the similarity evaluation is completed, and the processes from S204 to S212 are repeated. When the similarity evaluation process is completed for all moire regions, the similarity calculation process is terminated.

  In addition, while repeating the processing from S204 to S212, in S208, the similarity calculation result in each pixel is updated so that the similarity calculation result between the target pixel and the moire region having the highest similarity is stored.

  In this embodiment, the similarity between the target pixel and the adjacent moire area is calculated based on the subject distance, the color signal, and the frequency component, but the similarity is calculated with reference to other feature amounts. It doesn't matter.

  Further, the calculation of the degree of similarity with the adjacent moire area may be performed for each pixel, or the input image may be calculated based on the feature amount for each local area by dividing the input image into blocks.

  Finally, with reference to FIG. 4, the operation of the moire detection and reduction processing in the present embodiment will be described.

  FIG. 4A shows an example of a shooting region shot in this embodiment. An angle of view captured by the first imaging unit is 301, and an angle of view captured by the second imaging unit is 302.

  Of the continuous subject area 303 included in the angle of view 301, the area 304 can be imaged together by the first imaging means and the second imaging means.

  FIG. 4B is a schematic diagram illustrating an image output from the first imaging unit, and an image region 401 corresponds to the subject region 303 in FIG. Since the optical image of the subject area 303 includes a signal having a frequency higher than the Nyquist frequency determined by the pixel pitch of the image sensor, moire occurs in the corresponding image area 401. The area 402 (common area) is also included in the image output from the second imaging unit.

  FIG. 4C is a schematic diagram illustrating an image output from the second imaging unit, and an image region 501 corresponds to the subject region 304 in FIG. Although the subject area 304 is a part of the subject area 303, the second imaging unit performs imaging with a setting on the telephoto side than the first imaging unit, and therefore, is higher than the Nyquist frequency determined by the pixel pitch of the imaging element. The image is picked up as a low-frequency signal, and no moire occurs in the image area 501.

  FIG. 4D shows an image obtained by reducing the output image of the second imaging means by the geometric deformation means 103, and the image area 601 corresponds to the subject area 304 in FIG. 4A.

  Here, the image region 403 in FIG. 4B and the image region 601 in FIG. 4D are the same subject region, but the Nyquist frequency determined by the frequency component included in the subject image and the pixel pitch of the image sensor. Since the relative relationships of are different, the frequencies reproduced at the time of imaging are different. Therefore, in the region 601, the original frequency component of the subject can be reproduced, but in the region 403, moire occurs because the sampling theorem is not satisfied.

  Therefore, the moire region detecting unit 104 can detect whether or not moire has occurred by comparing the frequencies of the region 403 and the region 601. In the image output from the first imaging unit, FIG. ) Image area 403 is determined to be an area where moire occurs.

  Further, in the image output from the first imaging unit, the image area 401 is also a high-frequency subject area continuous with the area 403, so that moire occurs. However, the image corresponding to the image area 401 (non-common area) in FIG. 4B is not taken by the second imaging means because it is out of the shooting field angle on the telephoto side.

  Therefore, for the image region 401, the similarity calculation unit 105 calculates the image similarity with the image region 403, and performs moire correction processing by the moire correction unit based on the similarity with the moire region. The degree of similarity between the image area 403 determined to be a moire area and the image area 401 is based on subject distance information measured by distance measuring means (not shown), color signal dispersion, and luminance signal dispersion. Ask.

  As described above, in the imaging apparatus of the present invention, imaging is performed by independently controlling the optical system so that the frequency of moire generated in the same part of the subject image is different, and the moire area is detected from the difference between a plurality of images. It can be corrected. In addition, even when information for detecting a moiré component cannot be generated from the difference between the plurality of images, it is possible to correct the moiré based on the similarity to the moiré detection region and obtain a video signal with good image quality. it can.

  Next, a second embodiment of the imaging apparatus of the present invention will be described. The configuration of the image pickup apparatus of the second embodiment and the flowchart showing the processing flow are the same as those of the first embodiment, and thus the description thereof is omitted. In the second embodiment, the two-dimensional sampling direction when sampling the same subject image is different by 45 degrees relative to the first imaging unit 101 and the second imaging unit 102. The image pickup means is controlled.

  Specifically, imaging is performed by rotating the imaging device of the second imaging unit by 45 degrees so that the arrangement of the imaging device of the first imaging unit differs by 45 degrees around the optical axis.

  FIG. 5A shows an example of a shooting area shot in the second embodiment. In FIG. 5A, a rectangular area 701 is picked up by the first image pickup means, and a rectangular area 702 is picked up by the second image pickup means.

  The subject area 703 is an area including a higher frequency component than the Nyquist frequency determined by the pixel pitch in an oblique direction, and moire is generated.

  FIG. 5B shows an image output from the first imaging means, and has image areas 801 and 802 corresponding to the subject area 703 in FIG.

  FIG. 5C shows an image output from the second imaging means rotated 45 degrees to the right, and has an image area 901 corresponding to the subject area 703 in FIG. .

  In the second embodiment, the moire region detection unit 104 detects the frequency component for each direction in which the subject image is sampled two-dimensionally by the image sensor. As shown in FIG. 6, since the light receiving elements are discretely arranged in the horizontal and vertical directions in the imaging elements of the first imaging unit 101 and the second imaging unit 102, the moire region detection unit 104 uses the vertical direction. The frequency component in the direction and the frequency component in the horizontal direction are obtained, and the moire area is detected.

  Here, the oblique high-frequency component included in the region 703 of FIG. 5A includes both vertical and horizontal frequency components.

  Therefore, in the image of FIG. 5B output from the first image pickup means, moiré frequency components are detected in both the vertical and horizontal directions in the image regions 801 and 802.

  On the other hand, in the image of FIG. 5C output from the second image pickup means, the subject region 703 is sampled by the image pickup device rotated by 45 degrees, so that it is picked up as a vertical edge (horizontal stripe pattern). Is done. Therefore, when the moire region detecting means analyzes the frequency components in the vertical direction and the horizontal direction, the frequency component of moire is detected only in the vertical direction.

  For this reason, in the subject area 703, the areas that are imaged together by the first imaging means and the second imaging means are respectively added to the areas based on the difference between the moire frequencies in the vertical direction and the horizontal direction. Whether or not has occurred can be detected.

  On the other hand, in the image area 802 in FIG. 5B, since the corresponding image area does not exist in the image output from the second imaging means, the images captured by the first imaging means and the second imaging means. The moire area cannot be detected from the difference between the frequency components.

  Therefore, the similarity calculation unit 106 calculates the image similarity with the moire region 801, and performs moire correction processing with the moire correction unit based on the similarity with the moire region.

  As described above, also in the image pickup apparatus according to the second embodiment of the present invention, imaging is performed by independently controlling the sampling direction of the image pickup element so that the frequency of moire generated in the same portion of the subject image is different. The moire area can be detected and corrected from the difference between the images. In addition, even when information for detecting a moiré component cannot be generated from the difference between the plurality of images, it is possible to correct the moiré based on the similarity to the moiré detection region and obtain a video signal with good image quality. it can.

  As described above, when determining the moiré caused by the sampling of the sensor from the difference between a plurality of images simultaneously shot under different imaging conditions, the presence / absence of moiré is determined in an image area where the same subject does not exist. There was a problem that could not be done.

  On the other hand, the present invention determines a moire area from a plurality of images taken by changing optical characteristics or imaging sampling characteristics for the same subject, and based on the similarity to the moire area, Moire caused by sampling is reduced.

  Thus, according to the present invention, it is possible to generate (acquire) a video signal with good image quality in which the moire generated due to the imaging sampling characteristics is reduced (corrected) in all regions of the captured image.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  For example, when a software program that implements the functions of the above-described embodiments is supplied from a recording medium directly to a system or apparatus having a computer that can execute the program using wired / wireless communication, and the program is executed Are also included in the present invention.

  Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the present invention includes a computer program itself in which a procedure for realizing the functional processing of the present invention is described.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS. As a recording medium for supplying the program, for example, a magnetic recording medium such as a hard disk or a magnetic tape, an optical / magneto-optical storage medium, or a nonvolatile semiconductor memory may be used.

  As a program supply method, a computer program that forms the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program.

  The present invention can be suitably used for an imaging apparatus such as a compact digital camera, a single-lens reflex camera, and a video camera.

110 Image processing devices 110a and 110b Acquisition means 104 Moire area detection means 105 and 106 Similarity calculation means 107 and 108 Moire correction means

Claims (20)

Acquisition means for acquiring a plurality of images including a common area;
Determination means for determining a moire area in which the moire in the common area is generated among the plurality of images;
Calculating means for calculating a similarity with the moire region for a non-common region different from the common region among the plurality of images;
Correction means for correcting the moire for at least one of the plurality of images based on the moire area and the similarity;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the calculation unit calculates the similarity based on a feature amount of the non-common region and a feature amount of the moire region.   The image processing apparatus according to claim 2, wherein the feature amount is at least one of a color signal, a frequency component, and a subject distance.   The image processing apparatus according to claim 2, wherein the calculation unit calculates the similarity based on a difference in subject distance between the pixel of interest in the non-common area and the moire area.   The calculation means calculates the similarity when the difference in the subject distance is smaller than a first threshold, and does not calculate the similarity when the difference is larger than the first threshold. The image processing apparatus according to claim 4.   The image processing apparatus according to claim 2, wherein the calculation unit calculates the similarity based on a color signal difference between the target pixel in the non-common area and the moire area.   The calculating means calculates the similarity when the difference between the color signals is smaller than a second threshold, and does not calculate the similarity when larger than the second threshold. The image processing apparatus according to claim 6. The calculating means calculates a square value of the difference between the color signals as the similarity;
The image processing apparatus according to claim 6, wherein the calculated similarity is stored in a storage unit in association with the moire area.   The image processing apparatus according to claim 2, wherein the calculation unit calculates the similarity based on a frequency component difference between the target pixel in the non-common area and the moire area.   The calculating means calculates the similarity when the difference between the frequency components is smaller than a third threshold, and does not calculate the similarity when larger than the third threshold. The image processing apparatus according to claim 9. The calculation means calculates a square value of the difference between the frequency components as the similarity,
The image processing apparatus according to claim 9 or 10, wherein the calculated similarity is stored in a storage unit in association with the moire area.   The determination unit determines the common area where the moire area can be determined and the non-common area where the moire area cannot be determined from the difference between the plurality of images. The image processing apparatus according to any one of Items 11 to 11.   The image processing apparatus according to claim 1, wherein the correction unit changes the degree of correction according to the similarity in the non-common area.   The image processing apparatus according to claim 1, wherein the plurality of images are images obtained by imaging under different imaging conditions.   The different imaging conditions are at least one of a different focal length of the optical system and a different two-dimensional sampling direction when the imaging unit samples the subject image. 14. The image processing apparatus according to 14. An imaging means having an optical system and an imaging device;
An image pickup apparatus comprising: the image processing apparatus according to claim 1.   The imaging apparatus according to claim 16, wherein the imaging apparatus is a multi-eye imaging apparatus including a plurality of imaging units. Obtaining a plurality of images including a common area;
A step of determining a moiré area where moiré in the common area occurs among the plurality of images;
Calculating a similarity with the moire region for a non-common region different from the common region among the plurality of images;
Correcting the moire for at least one of the plurality of images based on the moire area and the similarity; and
An image processing method comprising:   A computer-executable program in which the steps of the image processing method according to claim 18 are described.   A computer-readable storage medium storing a program for causing a computer to execute the steps of the image processing method according to claim 18.