JP2015130076A - Image processing apparatus, imaging device, imaging system, image processing method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that can reduce deformation or emphasis of noise due to an imaging element in restoration processing of an image deteriorated due to an optical system.SOLUTION: An image processing apparatus includes an image restoration unit that performs image restoration processing on a first image obtained through an optical system to create a second image, a composition ratio calculation unit that calculates a composition ratio of the first image and the second image according at least to information based on the second image, and a composing unit that creates a composite image of the first image and the second image according to the composition ratio.

Description

  The present invention relates to an image processing apparatus that recovers a captured image.

  Patent Document 1 discloses an image processing method for recovering an image deteriorated due to lens aberration using an image recovery filter generated or selected based on an optical transfer function of an imaging system.

JP 2011-123589 A

  However, the image data imaged by the imaging system includes random noise caused by the imaging element. This random noise is not noise caused by lens aberration. Therefore, when an image restoration filter generated or selected based on the optical transfer function is applied, the deteriorated image of the subject is restored, but the random noise component is deformed or emphasized (noise characteristics are changed).

  Therefore, the present invention provides an image processing apparatus, an imaging apparatus, an imaging system, an image processing method, a program, and a storage capable of reducing the deformation or enhancement of noise caused by an imaging element in a recovery process of a deteriorated image caused by an optical system. Provide media.

  An image processing apparatus according to an aspect of the present invention includes an image recovery unit that generates a second image by performing an image recovery process on a first image obtained via an optical system, and at least the second A combination ratio calculation unit that calculates a combination ratio of the first image and the second image according to information based on the image of the first image, and the first image and the second image according to the combination ratio. And a synthesizing unit that generates the synthesized image.

  An imaging device according to another aspect of the present invention includes an imaging device that photoelectrically converts an optical image obtained through an optical system to obtain an image signal, and an image restoration process for the first image based on the image signal An image restoration unit that generates a second image by performing a calculation, and a combination ratio calculation unit that calculates a combination ratio of the first image and the second image according to information based on at least the second image And a combining unit that generates a combined image of the first image and the second image according to the combining ratio.

  An imaging system as another aspect of the present invention includes a lens device including an optical system and the imaging device.

  An image processing method according to another aspect of the present invention includes an image recovery step of generating a second image by performing an image recovery process on a first image obtained through an optical system, and at least the first A composite ratio calculating step of calculating a composite ratio between the first image and the second image according to information based on the second image; and the first image and the second image according to the composite ratio. Generating a composite image.

  A program according to another aspect of the present invention causes a computer to execute the image processing method.

  A storage medium according to another aspect of the present invention stores the program.

  Other objects and features of the invention are described in the following embodiments.

  According to the present invention, an image processing apparatus, an imaging apparatus, an imaging system, an image processing method, a program, and an image processing apparatus capable of reducing the deformation or enhancement of noise caused by an imaging element in the recovery process of a deteriorated image caused by an optical system, and A storage medium can be provided.

1 is a block diagram of an imaging apparatus that includes an image processing apparatus according to Embodiment 1. FIG. It is a figure which shows the change of the image before and behind an image restoration process. It is a figure which shows the difference absolute value of a pre-processing image and a recovery image. 6 is a block diagram of an imaging apparatus including an image processing apparatus according to Embodiment 2. FIG. In Embodiment 2, it is a figure which shows the reduction | decrease of a color blur by an image restoration process, and the deformation | transformation of a noise. FIG. 10 is a block diagram of an imaging apparatus provided with an image processing apparatus according to a third embodiment. 10 is a block diagram of a composition ratio calculation unit in Embodiment 3. FIG. It is a block diagram of an edge detection unit and an explanatory diagram of filter coefficients in the third embodiment. 10 is a graph showing a relationship between an input pixel value and an output pixel value of a coring unit in the third embodiment. 10 is a graph illustrating a relationship between an input pixel value and an output pixel value of a limiter unit in the third embodiment. 6 is a graph illustrating a combination ratio of a pixel value p of a preprocessed image and a pixel value r of a restored image in the first embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Embodiment 1]
First, an image processing apparatus and an imaging apparatus according to Embodiment 1 of the present invention will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of an imaging device 10 including an image processing device 15 according to the present embodiment.

  In FIG. 1, reference numeral 100 denotes an optical system (photographing optical system). In the present embodiment, the imaging device 10 is integrally configured with the optical system 100, but is not limited thereto. The present embodiment can also be applied to an imaging system including an imaging apparatus main body including the image processing device 15 and an optical system 100 (lens device) that can be attached to and detached from the imaging apparatus main body.

  Reference numeral 101 denotes an image sensor including a CMOS sensor or a CCD sensor. The image sensor 101 photoelectrically converts a subject image (optical image) obtained via the optical system 100 and outputs an image signal. Reference numeral 102 denotes an A / D conversion unit. The A / D converter 102 converts the image signal (analog image signal) output from the image sensor 101 into a digital image signal. Reference numeral 103 denotes a preprocessing unit. The preprocessing unit 103 performs preprocessing on the digital image signal output from the A / D conversion unit 102.

  Reference numeral 104 denotes an image recovery unit. The image recovery unit 104 performs image recovery processing on the output signal (preprocessed image) from the preprocessing unit 103. That is, the image restoration unit 104 generates a restored image (second image) by performing image restoration processing on the preprocessed image (first image) obtained via the optical system 100. As a result, a second image (recovered image) is generated from the first image (degraded image) degraded by the aberration of the optical system 100.

  Reference numeral 105 denotes a composition ratio calculation unit. The composition ratio calculation unit 105 calculates the composition ratio between the output signal from the preprocessing unit 103 (preprocessed image, that is, the first image) and the output signal from the image restoration unit 104 (recovered image, that is, the second image). calculate. Reference numeral 106 denotes a synthesis unit. The combining unit 106 outputs an output signal (first image) from the preprocessing unit 103 and an output signal (second image) from the image restoration unit 104 in accordance with the combination ratio calculated by the combination ratio calculation unit 105. Are combined to generate a composite image.

  Reference numeral 107 denotes a signal processing unit. The signal processing unit 107 performs predetermined signal processing on the output signal (composite image) from the synthesis unit 106 and outputs the signal after signal processing to the output terminal 108. In the present embodiment, the imaging device 10 includes an image processing device 15, and the image processing device 15 includes an image restoration unit 104, a composition ratio calculation unit 105, a composition unit 106, and a signal processing unit 107.

  The optical system 100 includes one or more lenses, a diaphragm, and the like, and includes at least optical aberrations (distortion aberration, axial chromatic aberration, lateral chromatic aberration, etc.). The imaging element 101 is an element that forms an image projected through the optical system 100, and the captured image (image formation image) includes noise (random noise). The A / D conversion unit 102 quantizes a captured image including random noise obtained from the image sensor 101 and converts it into a digital image signal. The preprocessing unit 103 performs preprocessing on the digital image signal obtained from the A / D conversion unit 102. The preprocessing is correction processing required before image restoration processing, such as defective pixel correction of the image sensor 101 and peripheral light amount drop correction. However, in the present embodiment, the preprocessing unit 103 is not necessary when correction processing (preprocessing) is not required before performing the image restoration processing.

  The image recovery unit 104 performs image recovery processing on the output signal (preprocessed image) from the preprocessing unit 103. The image restoration process is a process for reducing aberration caused by the lens (optical system 100). As a specific image restoration method, various known examples are known as disclosed in Patent Document 1. As a result of the image restoration process, for example, a result as shown in FIG. 2 is obtained. FIG. 2 is a diagram illustrating changes in the image before and after the image restoration process. In the photographed image (preprocessed image), blurs 201 and 203 due to aberration occur around the edge 202 of the subject, as shown as a part 204 of the preprocessed image. The preprocessed image portion 204 also includes noise 200. When image restoration processing is performed on such a photographed image (pre-processed image), bleeding around the recovered edge 207 is reduced as shown as a portion 206 of the restored image. However, the shape of the noise 200 is deformed like the deformed noise 205 due to the effect of the image restoration filter.

  The composition ratio calculation unit 105 compares the preprocessed image and the restored image, and calculates a composition ratio. This composition ratio corresponds to the mixing ratio of the preprocessed image and the restored image by the composition unit 106 at the subsequent stage. Here, with reference to FIG. 3, a method of calculating the composition ratio will be described. FIG. 3 is a diagram showing the absolute difference between the preprocessed image and the restored image. In FIG. 3, a partial area obtained by cutting out a high image height area (an area far from the center of the optical axis of the lens) where a lot of aberration occurs is taken up, but the processing target is applied to the entire area of the image. Also good.

  In FIG. 3, an absolute difference value 302 is an absolute difference value between the part 300 of the preprocessed image and the part 301 of the recovered image, and indicates edge information. The composition ratio calculation unit 105 determines a composition ratio based on the pixel value corresponding to the absolute difference value 302.

  When the pixel value corresponding to the absolute difference value 302 is equal to or greater than the first threshold value, the synthesizing unit 106 outputs the pixel value of the recovered image at the pixel position. In addition, when the pixel value corresponding to the absolute difference value 302 is smaller than the first threshold value and larger than the second threshold value, the synthesizing unit 106 determines the pixel value of the preprocessed image and the restored image at the pixel position. The weighted average value with the pixel value is output. When the pixel value corresponding to the absolute difference value 302 is equal to or smaller than the second threshold value, the synthesizing unit 106 outputs the pixel value of the preprocessed image at the pixel position. Hereinafter, the calculation method by the synthesis unit 106 will be described in detail using mathematical expressions.

  The pixel value of the preprocessed image is p1, the pixel value of the restored image is r1, the first threshold is T1, the second threshold is T2, the difference absolute value is d1, and the output pixel value (the output signal from the synthesis unit 106). Let o. Further, it is necessary to satisfy T1> T2 as a threshold parameter setting condition.

  First, the absolute difference value d1 is obtained by the calculation of the following equation (1).

d1 = | p1-r1 | (1)
The output pixel value o is calculated by the calculation of the following equation (2) using the difference absolute value d1 calculated by the equation (1) as a weight and using the threshold parameters T1 and T2.

o = r1 (when d1 ≧ T1)
o = {r1 · (d1−T2) + p1 · (T1−d1)} / (T1−T2) (when T1>d1> T2)
o = p1 (when T2 ≧ d1) (2)
FIG. 11 is a graph showing a composition ratio between the pixel value p1 of the preprocessed image and the pixel value r1 of the restored image, and corresponds to the relationship of Expression (2). 11A shows the relationship between the composite ratio of the pixel value r1 of the restored image and the edge information (difference absolute value d1), and FIG. 11B shows the pixel value p1 of the preprocessed image and the edge information (difference absolute value d1). The relationship is shown respectively. An output image constituted by the output pixel value o becomes a synthesis result by the synthesis unit 106. The signal processing unit 107 performs signal processing such as gamma correction and sharpness correction necessary for picture creation (image formation) on the synthesis result obtained by the synthesis unit 106. An output signal from the signal processing unit 107 is output to the output terminal 108.

  As described above, in the present embodiment, the composition ratio calculation unit 105 calculates the first image and the first image based on the information included in the first image (preprocessed image, that is, the captured image) or the second image (recovered image). A composite ratio with the second image is calculated. Preferably, the information included in the first image or the second image is edge information.

  Preferably, the composition ratio calculation unit 105 calculates a composition ratio based on the absolute difference value between the first image and the second image. More preferably, the composition unit 106 generates a composite image using the second image (recovered image) at the first pixel position where the absolute difference value is equal to or greater than the first threshold (d1 ≧ T1). Further, the synthesizing unit 106 displays the first image (preprocessed image, that is, the captured image) at the second pixel position where the absolute difference value is equal to or smaller than the second threshold value (T2 ≧ d1) smaller than the first threshold value. To generate a composite image. The synthesizing unit 106 also includes the first image and the first image at the third pixel position where the absolute difference value is smaller than the first threshold value and larger than the second threshold value (T1> d1> T2 is established). A composite image is generated using both of the two images. More preferably, when the absolute difference value is closer to the second threshold value than the first threshold value at the third pixel position, the synthesizing unit 106 increases the synthesis ratio of the first image as compared to the second image. That is, image composition is performed mainly using the first image. On the other hand, if the absolute difference value is closer to the first threshold value than the second threshold value, the composition ratio of the second image is increased as compared to the first image. That is, image composition is performed mainly using the second image.

  According to the present embodiment, an image deteriorated due to the optical system 100 is appropriately recovered for an area where the recovery effect by the image recovery unit 104 is large, and an area where the recovery effect by the image recovery unit 104 is small is recovered. Can obtain an output image with a natural noise shape.

[Embodiment 2]
Next, an image processing apparatus and an imaging apparatus according to Embodiment 2 of the present invention will be described with reference to FIG. FIG. 4 is a block diagram illustrating a configuration of the imaging device 40 including the image processing device 45 according to the present embodiment. 4, the optical system 400, the image sensor 401, the A / D conversion 402, the preprocessing unit 403, and the image restoration unit 404 are the optical system 100, the image sensor 101, the A / D conversion unit 102, and the preprocessing unit shown in FIG. 103 and the image recovery unit 104.

  Reference numeral 405 denotes a second signal processing unit. The second signal processing unit 405 performs signal processing such as gamma correction and sharpness correction on the output signal from the image recovery unit 404 (recovered image, that is, the second image). Reference numeral 406 denotes a first signal processing unit. The first signal processing unit 406 performs signal processing equivalent to the second signal processing unit 405 on the output signal (preprocessed image, that is, the first image) from the preprocessing unit 403.

  Reference numeral 407 denotes a composite ratio calculation unit. The synthesis ratio calculation unit 407 includes a second output signal (output signal based on the second image) from the second signal processing unit 405 and a first output signal (first output signal from the first signal processing unit 406). The composite ratio with the output signal based on the image is calculated. That is, the synthesis ratio calculation unit 407 uses the first output signal from the first signal processing unit 406 and the second output signal from the second signal processing unit 405 to perform the first image and the second output signal. The composition ratio with the image is calculated.

  Reference numeral 408 denotes a synthesis unit. The combining unit 408 combines the output signal from the second signal processing unit 405 and the output signal from the first signal processing unit 406 based on the combination ratio calculated by the combination ratio calculation unit 407. The combined signal is output to the output terminal 409. In the present embodiment, the imaging device 40 includes an image processing device 45, and the image processing device 45 includes an image restoration unit 404, a second signal processing unit 405, a first signal processing unit 406, a synthesis ratio calculation unit 407, and The combining unit 408 is configured.

  The second signal processing unit 405 performs signal processing such as gamma correction and sharpness correction necessary for picture creation (image formation) on the image recovered by the image recovery unit 404 (recovered image). The first signal processing unit 406 performs signal processing equivalent to the second signal processing unit 405 on the output signal (preprocessed image) from the preprocessing unit 403. As described above, in the present embodiment, after the signal processing (after image formation), the synthesis ratio is calculated and the signals are synthesized. In the present embodiment, the second output signal from the second signal processing unit 405 may be referred to as a restored image, and the first output signal from the first signal processing unit 406 may be referred to as a pre-recovery image.

  Next, the synthesis ratio calculation unit 407 will be described with reference to FIG. FIG. 5 is a diagram illustrating color blur reduction and noise deformation by image restoration processing. As shown in FIG. 5, blurs 501 and 503 due to aberration occur around the edge 502 of the subject, as shown as a part 504 of the pre-recovery image, in the captured image (pre-recovery image). When image restoration processing is performed on such a pre-recovery image, color blurs 501 and 503 due to aberration are reduced as shown as a portion 506 of the restored image. However, color blur occurs around the noise 500 due to the image restoration process, and the noise 500 becomes a deformed noise 505.

  The composition ratio calculation unit 407 compares the pre-recovery image and the recovered image to calculate a composition ratio. Hereinafter, a specific calculation method of the synthesis ratio will be described using mathematical expressions. When the pixel value is indicated by the three primary colors (R, G, B) of light, the saturation S can be expressed as the following equation (3), for example.

M = max (R, G, B)
m = min (R, G, B)
S = 0 (when M = 0)
S = (M−m) / M (when M ≠ 0) (3)
In Expression (3), max () is a function for obtaining the maximum value of a given element, and min () is a function for obtaining the minimum value of the given element. The pre-recovery image and the recovered image are overlapped, and the saturation S of the pixel value at the same position is compared. At this time, the composition ratio calculation unit 407 calculates the composition ratio so that the pixel value with the lower saturation S of the pre-recovery image and the recovered image is selected.

  Assuming that the saturation of the pre-recovery image is S1 and the saturation of the recovered image is S2, the composition ratio d2 is calculated as, for example, the following equation (4).

d2 = 0 (S1> S2)
d2 = 1 (S2 ≧ S1) (4)
The synthesizing unit 408 synthesizes the restored image and the pre-recovery image using the synthesis ratio d2 obtained individually for each pixel. The pixel value of the restored image is r2, the pixel value of the pre-recovery image is p2, and the output pixel value (output signal from the synthesis unit 408) is o. At this time, the output pixel value o (composition result) is calculated, for example, as in the following equation (5).

o = p2 · (1−d2) + r2 · d2 (5)
The synthesizing unit 408 outputs an image composed of the output pixel value o calculated by Expression (5) to the output terminal 409.

  Thus, in the present embodiment, the composition ratio calculation unit 407 calculates the composition ratio based on the respective saturations of the first output signal (pre-recovery image) and the second output signal (post-recovery image). Preferably, the synthesis unit 408 generates a synthesized image using the first output signal at a fourth pixel position where the saturation of the first output signal is lower than the saturation of the second output signal. The synthesizing unit 408 generates a synthesized image using the second output signal at the fifth pixel position where the saturation of the second output signal is lower than the saturation of the first output signal. It should be noted that the composition ratio calculation method of the composition ratio calculation unit 407 in the present embodiment and the composition ratio calculation method of the composition ratio calculation unit 105 in the first embodiment can be interchanged.

  According to the present embodiment, an output image having a natural noise shape is reduced for a region where coloring has occurred due to image degradation caused by the optical system 100 and a region other than that is colored. Can be obtained.

[Embodiment 3]
Next, an image processing apparatus according to the third embodiment of the present invention will be described with reference to FIG. FIG. 6 is a block diagram showing the configuration of the image processing apparatus according to this embodiment.

  Next, with reference to FIG. 6, an image processing apparatus and an imaging apparatus according to Embodiment 3 of the present invention will be described. FIG. 6 is a block diagram illustrating a configuration of the imaging device 60 including the image processing device 65 according to the present embodiment. In FIG. 6, the optical system 600, the image sensor 601, the A / D conversion 602, the preprocessing unit 603, and the image restoration unit 604 are the optical system 100, the image sensor 101, the A / D conversion unit 102, and the preprocessing unit in FIG. 1. 103 and the image recovery unit 104. The synthesizing unit 606 and the signal processing unit 607 are the same as the synthesizing unit 106 and the signal processing unit 107 in FIG. That is, the imaging apparatus 60 in FIG. 6 differs from the imaging apparatus 10 in FIG. 1 only in the composition ratio calculation unit 605.

  With reference to FIG. 7, the composition ratio calculation unit 605 in the present embodiment will be described. FIG. 7 is a block diagram of the composition ratio calculation unit 605. In the composition ratio calculation unit 605, 700 is an input terminal for inputting an output signal (recovered image) from the image recovery unit 604. Reference numeral 701 denotes an edge detection unit that detects an edge of a recovered image and obtains an edge signal. A coring unit 702 removes noise included in the edge signal. Reference numeral 703 denotes a limiter unit that performs upper limit clipping of the edge signal from which noise has been removed. Reference numeral 704 denotes a MAX filter unit that applies a MAX filter to the output signal from the limiter unit 703. Reference numeral 705 denotes an output terminal to which an output signal from the MAX filter unit 704 is output.

  The edge detection unit 701 performs edge detection and determines whether each pixel position is an edge portion or a flat portion. Then, the edge detection unit 701 outputs a first value (large value) to the pixel position determined to be an edge part. On the other hand, the edge detection unit 701 outputs a second value (small value) smaller than the first value at the pixel position determined to be a flat part. Although there are various configurations as the edge detection unit 701, the edge detection unit 701 shown in FIG. 8 is adopted in this embodiment.

  FIG. 8A is a block diagram of the edge detection unit 701 and an explanatory diagram of filter coefficients in the present embodiment. In FIG. 8A, reference numeral 800 denotes an input terminal for inputting an input image (recovered image). Reference numeral 801 denotes a horizontal Sobel filter unit. The horizontal Sobel filter unit 801 applies a horizontal Sobel filter to the input image to detect horizontal edges. Reference numeral 802 denotes a vertical Sobel filter unit. The vertical Sobel filter unit 802 applies a vertical Sobel filter to the input image, and detects vertical edges.

  Reference numeral 803 denotes an absolute value portion for calculating the absolute value of the horizontal edge. Reference numeral 804 denotes an absolute value portion for calculating the absolute value of the vertical edge. Reference numeral 805 denotes an adder that adds the output signals from the absolute value units 803 and 804 (the absolute value of the horizontal edge and the absolute value of the vertical edge). An output signal (addition signal) from the adder 805 is output to the output terminal 806.

  The horizontal Sobel filter unit 801 applies a 3 × 3 pixel spatial FIR filter centered on the pixel of interest. As a filter coefficient of the horizontal Sobel filter unit 801, for example, a value shown in FIG. 8B is used. The vertical Sobel filter unit 802 applies a 3 × 3 pixel spatial FIR filter around the pixel of interest. As a filter coefficient of the vertical Sobel filter unit 802, for example, a value shown in FIG. 8C is used. The value (edge information) output from the output terminal 805 indicates that the larger the value is, the closer to the edge portion (closer to the edge portion), and the smaller the value, the flatter portion (close to the flat portion). Show.

  The coring unit 702 subtracts a predetermined constant and replaces the negative pixel value with 0. FIG. 9 is a graph showing the relationship between the input pixel value and the output pixel value of the coring unit 702. T in FIG. 9 is a predetermined constant that decreases. The constant T is determined by the balance between the size of the noise component to be removed and the size of the edge to be left. The limiter unit 703 clips a value exceeding the upper limit. FIG. 10 is a graph showing the relationship between the input pixel value and the output pixel value of the limiter unit 703.

  The MAX filter unit 704 refers to a range of n pixels × n pixels centering on the pixel of interest, and uses the value that is the maximum value within the range as the output of the pixel of interest position. As the value n, a value obtained by converting the width of blur due to image degradation of interest into the number of pixels is used. For example, n is 3 · 2 + 1 = 7 when performing image restoration of lens aberration in which bleeding of up to 3 pixels occurs. An output signal from the MAX filter unit 704 is output to the output terminal 705.

  As described above, in the present embodiment, the composition ratio calculation unit 605 includes the edge detection unit 701 that detects edge information of the recovered image (second image). Then, the synthesis unit 606 calculates a synthesis ratio between the preprocessed image (first image) and the restored image (second image) based on the edge information detected by the edge detection unit 701. The synthesizing unit 606 may calculate the synthesis ratio of the preprocessed image and the restored image according to FIG. 11 with the edge information detected by the edge detecting unit 701 as d1.

  According to the present embodiment, the image degradation caused by the optical system 100 is recovered for the edge portion, and a natural noise-shaped output image can be obtained for the flat portion. The first signal processing unit 406 according to the second embodiment is provided between the preprocessing unit 603 and the synthesis unit 606 according to the present embodiment, and the second signal processing unit 405 according to the second embodiment is used as the image according to the present embodiment. You may provide between the recovery | restoration part 604 and the synthetic | combination ratio calculation part 605. FIG. In this case, the combination ratio calculation unit 605 detects edge information using the second output signal from the second signal processing unit 405.

[Other Embodiments]
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed. In this case, a computer-executable program describing the procedure of the imaging apparatus control method and a storage medium storing the program constitute the present invention.

  According to each embodiment, an image processing device, an imaging device, an imaging system, and an image that can reduce noise deformation or enhancement (change in noise characteristics) caused by an image sensor in a recovery process of a deteriorated image caused by an optical system. A processing method, a program, and a storage medium can be provided.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

15 Image processing apparatus 104 Image restoration unit 105 Composition ratio calculation unit 106 Composition unit

Claims (20)

An image recovery unit that generates a second image by performing an image recovery process on the first image obtained via the optical system;
A synthesis ratio calculation unit that calculates a synthesis ratio of the first image and the second image according to information based on at least the second image;
An image processing apparatus comprising: a combining unit that generates a combined image of the first image and the second image in accordance with the combining ratio.   The image processing apparatus according to claim 1, wherein the synthesis ratio calculation unit calculates the synthesis ratio according to edge information that is an absolute difference value between the first image and the second image. .   The image processing apparatus according to claim 1, wherein the synthesis ratio calculation unit calculates the synthesis ratio according to edge information detected from the second image.   The combination ratio calculation unit is configured to output the first image with respect to the first image at a first pixel position where the edge information is equal to or greater than a first threshold value than at a second pixel position where the edge information is smaller than the first threshold value. The image processing apparatus according to claim 2, wherein the composition ratio is calculated so that the composition ratio of the second image is high. The synthesis unit is
Generating the composite image using the second image at a first pixel position where the edge information is greater than or equal to a first threshold;
Generating the composite image using the first image at a second pixel position where the edge information is equal to or less than a second threshold value that is smaller than the first threshold value;
In the third pixel position where the edge information is smaller than the first threshold and larger than the second threshold, the composite image is obtained using both the first image and the second image. The image processing apparatus according to claim 2, wherein the image processing apparatus generates the image processing apparatus. In the third pixel position, the synthesis unit
When the edge information is closer to the second threshold than the first threshold, the composition ratio of the first image is increased than the second image;
6. The image processing according to claim 5, wherein when the edge information is closer to the first threshold value than the second threshold value, a composition ratio of the second image is higher than that of the first image. apparatus.   The image processing apparatus according to claim 1, wherein the composition ratio calculation unit calculates the composition ratio based on the saturation of each of the first image and the second image. The synthesis unit is
Generating the composite image using the first image at a fourth pixel position where the saturation of the first image is lower than the saturation of the second image;
The combined image is generated using the second image at a fifth pixel position where the saturation of the second image is lower than the saturation of the first image. The image processing apparatus described. A first signal processing unit that performs signal processing on the first image;
A second signal processing unit that performs signal processing on the second image,
The synthesis ratio calculation unit uses the first output signal from the first signal processing unit and the second output signal from the second signal processing unit to use the first image and the second image. The image processing apparatus according to claim 1, wherein the composition ratio with the image is calculated.   10. The image according to claim 9, wherein the synthesis ratio calculation unit calculates the synthesis ratio according to edge information that is a difference absolute value between the first output signal and the second output signal. 10. Processing equipment.   The image processing apparatus according to claim 9, wherein the synthesis ratio calculation unit calculates the synthesis ratio according to edge information detected from the second output signal.   The synthesis ratio calculation unit may output the first output at a first pixel position where the edge information is equal to or greater than a first threshold value than at a second pixel position where the edge information is smaller than the first threshold value. The image processing apparatus according to claim 10, wherein the synthesis ratio is calculated so that a synthesis ratio of the second output signal to the signal is high.   The image processing apparatus according to claim 9, wherein the synthesis ratio calculation unit calculates the synthesis ratio based on a saturation of each of the first output signal and the second output signal. The synthesis unit is
Generating the composite image using the first output signal at a fourth pixel position where the saturation of the first output signal is lower than the saturation of the second output signal;
The composite image is generated using the second output signal at a fifth pixel position where the saturation of the second output signal is lower than the saturation of the first output signal. Item 14. The image processing apparatus according to Item 13.   15. The image restoration unit according to claim 1, wherein the image restoration unit generates the second image by performing the image restoration process on the first image deteriorated due to an aberration of the optical system. The image processing apparatus according to any one of the above. An image sensor that photoelectrically converts an optical image obtained through an optical system to obtain an image signal;
An image recovery unit that generates a second image by performing an image recovery process on the first image based on the image signal;
A synthesis ratio calculation unit that calculates a synthesis ratio of the first image and the second image according to information based on at least the second image;
An image pickup apparatus comprising: a combining unit that generates a combined image of the first image and the second image according to the combining ratio. A lens device equipped with an optical system;
An imaging system comprising: the imaging device according to claim 16. An image recovery step of generating a second image by performing an image recovery process on the first image obtained via the optical system;
A composition ratio calculating step of calculating a composition ratio between the first image and the second image according to information based on at least the second image;
An image processing method comprising: a combining step of generating a combined image of the first image and the second image in accordance with the combining ratio.   A program configured to cause a computer to execute the image processing method according to claim 18.   A storage medium storing the program according to claim 19.