JP2023111637A - Image processing device and method and imaging apparatus - Google Patents

Abstract

To generate a high-resolution image having high color reproducibility from a plurality of images captured by using a plurality of imaging elements that differ in color reproducibility.SOLUTION: An image processing device includes acquisition means for acquiring a first image having first resolution and first color reproducibility and a second image having second resolution lower than the first resolution and second color reproducibility higher than the first color reproducibility, obtained by imaging one subject by a plurality of imaging means and synthesis means for generating a synthetic image by synthesizing the first image and the second image with a synthesis ratio corresponding to brightness and a frequency component calculated from the first image and the second image. The synthesis ratio of the color component of the second image when the brightness is first brightness is made higher than that when the brightness is second brightness brighter than the first brightness.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing apparatus and method for synthesizing a plurality of images captured using a plurality of imaging elements of different types, and an imaging apparatus.

Color reproduction and resolution of an image captured in a low-illuminance environment are important for purposes such as monitoring. A SPAD (Single Photon Avalanche Diode) sensor, which is one type of imaging sensor, is excellent in terms of low-illuminance performance (especially color reproduction) and high-speed response. However, an image captured by a SPAD sensor (SPAD image) generally has a lower resolution than an image captured by, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor (CMOS image).

In Patent Document 1, when a visible light image and a non-visible light (near-infrared light) image are combined, when it is determined that the environment is low illumination from imaging conditions such as shutter speed, the visible light luminance component and the non-visible light luminance component It is disclosed to increase the ratio of the latter when conducting the synthesis.

JP 2014-135627 A

However, the method of Patent Document 1 is a combination of luminance components of a visible light image having color information and an invisible light (near-infrared light) image having no color information. do not contribute.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to generate a high-resolution image with high color reproducibility from a plurality of images captured using a plurality of image sensors with different color reproducibility. and

In order to achieve the above object, an image processing apparatus of the present invention provides a first image having a first resolution and a first color reproducibility obtained by imaging the same subject with a plurality of imaging means; an acquisition means for acquiring a second image having a second resolution lower than the first resolution and a second color reproducibility higher than the first color reproducibility; and the first image; synthesizing means for synthesizing the second image and the second image at a synthesis ratio according to the brightness and frequency components obtained from the first image and the second image to generate a synthesized image; When the brightness is the first brightness, a synthesis ratio of the color components of the second image is set higher than when the brightness is the second brightness, which is brighter than the first brightness. .

According to the present invention configured as described above, a high-resolution image with high color reproducibility can be generated from a plurality of images captured using a plurality of imaging elements with different color reproducibility.

1 is a block diagram showing a schematic functional configuration of an imaging device according to an embodiment; FIG. FIG. 2 is a block diagram showing a partial configuration and processing of an image processing unit according to the first embodiment; 4 is a flowchart showing the flow of image synthesizing processing in an image processing unit according to the first embodiment; FIG. 4 is an explanatory diagram of a synthesis ratio of color components according to the first embodiment; FIG. 8 is a block diagram showing the partial configuration and processing of an image processing unit according to the second embodiment; FIG. 11 is a block diagram showing the partial configuration and processing of an image processing unit according to the third embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

<First Embodiment>
The control unit 101 is, for example, a CPU, and controls the operation of each block included in the image capturing apparatus 100 by reading an operation program for each block included in the image capturing apparatus 100 from the ROM 102, developing it in the RAM 103, and executing it. The ROM 102 is an electrically erasable/recordable non-volatile memory, and stores an operation program for each block included in the imaging apparatus 100 as well as parameters required for the operation of each block. A RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data output during operation of each block of the imaging apparatus 100 .

The imaging units 104-1 and 104-2 each include an optical system such as a lens and an imaging element, and each optical system is configured so that the same subject image is formed on each imaging element. The imaging unit 104-1 photoelectrically converts incident light to accumulate charges, and performs A/D conversion on electrical signals corresponding to the charges to obtain image signals. In this embodiment, the imaging device of the imaging unit 104-1 is a CMOS sensor, but may be a CCD (Charge Coupled Device) sensor or the like. It is assumed that the imaging device of the imaging unit 104-2 has lower resolution than the imaging device of the imaging unit 104-1 but has excellent color reproducibility at low illuminance. In the present embodiment, the SPAD sensor that counts the number of incident photons and obtains an image signal is used, but another type of imaging sensor that is excellent in color reproducibility at low illuminance may be used. A first image captured by the imaging unit 104-1 (hereinafter referred to as a “CMOS image” for convenience) and a second image captured by the imaging unit 104-2 (hereinafter referred to as a “ "SPAD image") is output to the RAM 103 and temporarily stored. Note that both the CMOS image and the SPAD image are images having Bayer array color information.

The image processing unit 105 applies various image processing to the CMOS image and SPAD image stored in the RAM 103, such as development processing such as demosaicing and white balance, color luminance conversion, and synthesis of the CMOS image and the SPAD image. .

The recording unit 106 is a detachable memory card or the like, and images processed by the image processing unit 105 are recorded as recording images via the RAM 103 .
A display unit 107 is a display device such as an LCD, and displays images recorded in the RAM 103 and the recording unit 106 and an operation user interface for receiving instructions from the user.

Next, image synthesizing processing performed by the image processing unit 105 according to the first embodiment will be described with reference to the block diagram of FIG. 2 and the flowchart of FIG.

FIG. 2 is a block diagram showing a partial configuration and processing of the image processing unit 105 of the first embodiment. The image processing unit 105 of the first embodiment includes reduction units 220 and 222 to 224, noise reduction units 231 to 234, same layer color component synthesis units 243 and 244, edge detection units 261 to 263, enlargement units 272 to 274, Different layer color component synthesizing units 281 to 283 are provided. The function of each part will be described later with reference to the flowchart of FIG.

In S301, development processing including demosaicing is performed on each of the CMOS image and the SPAD image to be synthesized, which are captured at the same time, to generate an RGB image, and then color luminance conversion is performed to generate a YUV image. Hereinafter, an image corresponding to the luminance component Y will be called a luminance component image, and an image corresponding to the color difference components U and V will be called a color component image.

In S302, for the color component image generated in S301, a reduced image for each layer is generated in order to hierarchically perform noise reduction processing at a plurality of resolutions as shown in FIG.

As an example, assuming that the resolution of a color component image of a CMOS image (hereinafter referred to as "CMOS color component image") 201 is 6000×4000, hierarchical processing is performed at four resolutions (hierarchies). First, the reduction unit 222 generates a CMOS color component image 202 of 3000×2000, which is half the resolution of the CMOS color component image 201 , from the CMOS color component image 201 . Next, the reduction unit 223 converts the 1/4 resolution 1500×1000 CMOS color component image 203 from the CMOS color component image 202, and the reduction unit 224 further converts the 1/8 resolution 750×500 CMOS color component image 203 into 1/8 resolution. A component image 204 is generated from the CMOS color component image 203 . As a result, the CMOS color component image is hierarchized into four hierarchies step by step.

On the other hand, it is assumed that the resolution of the color component image of the SPAD image (hereinafter referred to as “SPAD color component image”) 210 is 1800×1200, which is lower than the resolution of the CMOS color component image 201 . In that case, considering the effectiveness of the color component information, in this embodiment, the color component information of the SPAD image is used only in the layers below this resolution. Here, first, the reduction unit 220 generates a SPAD color component image 213 of 1500×1000, which is 1/4 the resolution of the CMOS color component image 201, from the SPAD color component image 210. FIG. Next, the reduction unit 224 generates from the SPAD color component image 213 a 750×500 SPAD color component image 214 that is ⅛ the resolution of the CMOS color component image 201 . As a result, the SPAD color component image is layered into two layers step by step.

Next, in S303, first, the noise reduction unit 234 performs noise reduction on the CMOS color component image 204 in the fourth layer, which is the lowest ⅛ resolution. For noise reduction, for example, an edge-preserving bilateral filter or the like is used. Edge Preserving Bilateral Filter The edge preserving bilateral filter has the effect of widening the effective reference range of the filter in hierarchical processing using reduced images. In general, the parameters of the noise reduction filter (kernel size, σ of Gaussian distribution, etc.) are set so that the noise reduction strength increases as the imaging sensitivity increases.

In S304, it is determined whether there is a SPAD color component image in the layer being processed. Since the fourth layer contains the SPAD color component image 214 generated in S302, the process proceeds to S305. In S305, the noise reduction unit 234 performs noise reduction on the SPAD color component image 214 as in S303.

In S306, the same layer color component synthesizing unit 244 calculates the synthesis ratio of the CMOS color component image 204 and the SPAD color component image 214 of the same fourth layer that have undergone noise reduction in S303 and S305.

At this time, the lower the illuminance, the greater the composition ratio of the SPAD color component, which has excellent low-illuminance color reproducibility. Here, the lower the illuminance, the greater the difference (absolute difference) between the CMOS color components and the SPAD color components due to the difference in color reproducibility. do. As an example, as shown in FIG. 4A, α=0 if the color component difference is equal to or less than the threshold thα0, and α=1 if the difference is equal to or greater than the threshold thα1, and linearly increases in between. Note that the lower the imaging sensitivity is, the lower the noise level is, and the smaller the difference in color components is. Therefore, the thresholds thα0 and thα1 may be decreased.

Further, the composite ratio α may be calculated for each pixel, or may be the same composite ratio for the entire image by taking the average of the entire image. Also, the difference between the color components may be normalized by, for example, dividing by the sum of the color components.

Further, the parameters of the noise reduction filters in S303 and S305 may be set so that the residual noise levels of the CMOS color components and the SPAD color components are approximately the same when the difference between the color components is taken. For example, due to the low noise characteristics of the SPAD sensor, the noise reduction strength in S303 for CMOS color components is increased.

In S307, using the synthesis ratio α calculated in S306, the same-hierarchy color component synthesizing unit 244 synthesizes the noise-reduced CMOS color component image 204 and the SPAD color component image 214 of the same fourth hierarchy to create a color component hierarchy. Image 254 is generated according to equation (1).

Color component hierarchical image = (1-α) x CMOS color component image + α x SPAD color component image (1)

In S308, it is determined whether the hierarchy being processed is the lowest hierarchy. Since the fourth hierarchy is the lowest hierarchy, the process proceeds to S312. In S312, it is determined whether the layer being processed is the highest layer. Since the fourth layer is not the highest layer, the process proceeds to S313.

In S313, the enlarging unit 274 expands the input 1/8 resolution (750×500) color component layer image 254 to 1/4 resolution (1500×1000), which is the resolution of the third layer one level higher. An enlarged color component layered image 254' is generated and output. Here, for example, bilinear interpolation or the like can be used for the enlargement processing.

After finishing the processing of the fourth layer as described above, the process returns to S303, and then the processing of the third layer of 1/4 resolution, which is one layer higher, is performed.

The processing of S303 to S307 is the same as the processing for the fourth layer described above, the noise reduction unit 233 performs noise reduction on the CMOS color component image 203 and the SPAD color component image 213, and the same layer color component synthesizing unit 243 A color component hierarchical image 253 is generated by synthesizing these images.

In S308, it is determined whether the hierarchy being processed is the lowest hierarchy. Since the third hierarchy is not the lowest hierarchy, the process proceeds to S309.

In S<b>309 , the edge detection unit 263 performs edge detection on the input color component hierarchical image 253 . An arbitrary edge detection filter such as a Sobel filter or a Laplacian filter can be used for edge detection.

In S310, the different layer color component synthesizing unit 283 calculates the synthesizing ratio of the color component layer image 253 of the third layer and the color component layer image 254' from the fourth layer one level below from the edge detection result of S309. calculate.

At this time, the larger the edge amount (for example, the absolute value of the application result of the edge detection filter) of the color component of the layer under processing, the larger the composition ratio β of the color component layer image 253 of the layer under processing. This is because the larger the edge amount, the higher the possibility that the subject included in the image is a subject that includes more high-frequency components (frequency components higher than a predetermined frequency). This is because it is considered that priority should be given to resolution in some cases. As an example, as shown in FIG. 4B, if the edge amount of the color component layer image in the layer being processed is equal to or less than the threshold thβ0, β=0, and if it is equal to or more than the threshold thβ1, β=1, and increases linearly in between. You may do so.

In S311, using the synthesis ratio β calculated in S310, the different layer color component synthesis unit 283 combines the color component layer image 253 of the third layer under processing and the color component layer image from the fourth layer one level below. 254' to generate a color component layered image 293 according to equation (2).

Color component layer image = β x Color component layer image of layer being processed + (1-β) x Color component layer image one layer below
…(2)

At this time, when the amount of edges is large (β is large), the composition ratio of the layer being processed with higher resolution increases, so that the sense of resolution is maintained. Also, when the edge amount is small (β is small), the synthesis ratio from the lower layer with lower resolution increases, so the noise reduction effect increases.

In S312, it is determined whether the current hierarchy is the highest hierarchy, and since the third hierarchy is not the highest hierarchy, the process proceeds to S313.

In S313, the enlarging unit 273 enlarges the color component layer image 293 from the input 1/4 resolution (1500×1000) to 1/2 resolution (3000×2000), which is the resolution of the second layer one level above. An enlarged color component hierarchical image 293' is generated and output.

When the processing of the third hierarchy is completed as described above, the process returns to S303, and then the processing of the second hierarchy of 1/2 resolution is performed.

In S<b>303 , the noise reduction unit 232 performs noise reduction on the CMOS color component image 202 . In S304, it is determined whether there is a SPAD color component image in the current layer, and since there is no SPAD color component image in the second layer, the process proceeds to S309.

In S309, the edge detection unit 262 performs edge detection of the CMOS color component image 202 that has undergone noise reduction in S303.

S310 to S313 are the same as the processing for the third hierarchy described above. The different layer color component synthesizing unit 282 combines the second layer CMOS color component image 202 noise-reduced by the noise reduction unit 233 and the color component layer image 293′ from the third layer one level below, through edge detection in S309. The results are used to synthesize and generate a color component layered image 292 .

Then, the enlarging unit 272 enlarges the input color component hierarchical image 292 of 1/2 resolution (3000×2000) to the first hierarchical resolution (6000×4000), which is one level higher. ' is generated and output.

After finishing the processing of the second layer as described above, the process returns to S303, and then the processing of the first layer is performed.

S303 to S311 are the same as the processing for the second layer described above. The different layer color component synthesizing unit 281 combines the CMOS color component image 201 of the first layer whose noise has been reduced by the noise reduction unit 231 and the color component layer image 292' from the second layer one level below to the edge detection unit 261. are synthesized using the edge detection results of , and a color component synthesized image 291 is generated.

In S312, it is determined whether the layer being processed is the highest layer. Since the first hierarchy is the highest hierarchy, the process proceeds to S314.

In S314, the color component composite image 291 is combined with the luminance component image of the same resolution (6000×4000). Here, the luminance component image to be combined may be a CMOS luminance component image, or may be a luminance component synthetic image generated by performing hierarchical composition processing of CMOS luminance components and SPAD luminance components in the same manner as the color components.

Hierarchical processing at a plurality of resolutions using reduced images can also be considered as processing for each frequency band. Therefore, the CMOS color component image and the SPAD color component image may be frequency-converted at the resolution of the CMOS image, and the CMOS color component and the SPAD color component may be combined more finely by frequency in the frequency domain.

FIG. 4(c) shows an image of the composition ratio of the CMOS color component image and the SPAD color component image in such a case. The difference (corresponding to illuminance) between the color component of the component image and the color component of the SPAD color component image is taken as the axis. In FIG. 4C, the higher the frequency (upper hierarchy) and the smaller the difference in color components (high illuminance), the higher the composition ratio of the CMOS color component image. Also, the larger the color component difference (low illuminance), the larger the composition ratio of the SPAD color component image.

As described above, according to the first embodiment, a color component image of a high-resolution image and a color component image of a low-resolution image excellent in low-illumination color reproduction are combined in consideration of the difference in resolution. , it is possible to achieve both low-illuminance color reproducibility and high resolution.

<Second embodiment>
Next, a second embodiment of the invention will be described.
In the example shown in FIG. 2 in the first embodiment described above, it was possible to use the color components of the SPAD image only at the resolutions of the third hierarchy and below among the four hierarchies. However, there may be cases where it is desirable to use the color components of the SPAD image up to a higher layer (frequency), such as a scene where color reproduction is important.

Here, since the SPAD sensor has the advantage of high-speed response, it is possible to capture a plurality of SPAD images while capturing one CMOS image. Therefore, in the present embodiment, a plurality of SPAD color component images are combined to increase the resolution, so that it is possible to cope with a case where it is desired to use the color components of the SPAD images in higher layers.

FIG. 5 is a block diagram showing a partial configuration and processing of the image processing unit 105 of the second embodiment. A difference from the image processing unit 105 of the first embodiment shown in FIG. In addition, the same reference numbers are given to the same parts as in the first embodiment, and the description thereof will be omitted as appropriate.

The resolution determination unit 501 determines whether the SPAD color component image is to be used up to a higher layer, and determines the resolution of the SPAD color component image.

Specifically, for example, the average saturation of an image is calculated from a SPAD image (RGB image), and the higher the saturation, the more important the color reproduction is considered to be in a scene. It is determined that it is a case where you want to use an image. Note that the saturation may be calculated from the CMOS image.

In addition, when an object such as a person is detected from a CMOS image, the smaller the number of pixels of the object, the more limited the information of the object in terms of resolution. It may be determined that it is a case in which the user wants to use the SPAD color component image.

Further, it may be determined that the SPAD sensor is more advantageous in terms of high-speed response as the motion amount (motion blur) of the subject increases, and that the SPAD color component image is desired to be used in higher layers.

It is also possible to make the determination in the process of hierarchical processing of color components. For example, the larger the difference between the CMOS color component image and the SPAD color component image in the same layer color component synthesizing units 243 and 244, the lower the quality of the CMOS color component image in the upper layer is expected. In that case, it may be determined that the SPAD color component image is desired to be used up to a higher layer. In particular, the larger the CMOS color component image in the higher hierarchy, the more desirable it is to use the SPAD color component image up to the higher hierarchy.

When the resolution determination unit 501 determines that the SPAD color component image is to be used up to the upper layer, the resolution for using the SPAD color component image is set to, for example, 3000×2000, which is the resolution of the second layer. decide. It should be noted that the resolution of a higher layer may be determined depending on the content of determination.

A super-resolution unit 502 generates a SPAD color component image having the resolution determined by the resolution determination unit 501 from a plurality of SPAD color component images. Here, specifically, from N SPAD color component images 510-1 to 510-N with a resolution of 1800×1200, a composite SPAD color component image 512 with a resolution of 3000×2000 in the second layer is generated.

Super-resolution processing, which synthesizes multiple images, generally consists of registration processing and reconstruction processing. An IBP (Iterative Back Projection) method or the like can be used.

The number N of SPAD color component images used for super-resolution is equal to or less than the number M of SPAD images that can be captured while capturing one CMOS image. Calculated by dividing. For example, when generating a second layer SPAD color component image from a SPAD color component image with a resolution of 1800×1200, round up 3000×2000/(1800×1200)≈2.8 and determine N=3.

If N<M, select the SPAD color component image to be used for super-resolution. At this time, it may simply be the top N sheets, or it may be selected so that the cluster flaw due to crosstalk of avalanche light emission, which tends to occur in the SPAD image, is not conspicuous.

For example, if cluster scratches (defective pixels of a predetermined size or larger) overlap the edge of the image, the correction mark will be noticeable even if the scratch correction is performed. ) to select an image. It is assumed that positional information of cluster flaws is already provided.

In addition, when aligning and synthesizing a plurality of SPAD color component images, cluster flaws are eliminated in the generated super-resolution SPAD color component image by not using the cluster flawed portions of each image. can be expected. At this time, it is preferable to select a plurality of SPAD color component images so as not to cause cluster flaws in all images in the aligned state.

The layer processing after the super-resolution unit 502 generates the synthesized SPAD color component image 512 is the same as the flowchart in FIG. The processing of the 2nd layer is the same as that of the 3rd layer.

That is, the noise reduction unit 232 performs noise reduction on the CMOS color component image 202 and the combined SPAD color component image 512 , and the same layer color component combining unit 542 combines these images to generate the color component layer image 552 . Further, the different layer color component synthesizing unit 282 combines the color component layer image 552 and the color component layer image 293' from the third layer one level below using the edge detection result of the color component layer image 552 by the edge detection unit 262. By synthesizing, a color component hierarchical image 592 is generated. Further, the enlarging unit 272 expands the input color component hierarchical image 592 of 1/2 resolution (3000×2000) to the first hierarchical resolution (6000×4000), which is one level higher, to obtain a color component hierarchical image 592′. is generated and output.

In the different layer color component synthesizing unit 281 of the first layer, the CMOS color component image 201 noise-reduced by the noise reduction unit 231 and the color component layer image 592' from the second layer one level below are combined with the edge detection unit 261. are combined using the edge detection results to generate a color component hierarchical image 591 .

In this embodiment, the SPAD color component images can be used even in the second layer, so that the color component layer images 592, 592', and 591 are obtained from the color component layer images 292, 292', and 291 of the first embodiment, respectively. Since the SPAD image has many color components, the color reproducibility is improved.

As described above, according to the second embodiment, a plurality of SPAD color component images are subjected to super-resolution processing to increase the resolution, making it possible to use the SPAD color component images in higher layers. Become.

It should be noted that super-resolution processing may be performed from a single image instead of a plurality of images. In that case, super-resolution processing by machine learning using deep learning or the like may be performed instead of classical enlargement processing such as bilinear interpolation.

<Third Embodiment>
Next, a third embodiment of the invention will be described.
In the third embodiment, by applying color conversion that brings the color components of the CMOS image closer to the color components of the SPAD image after the CMOS image is developed, Improves color familiarity and further improves color reproduction.

In the third embodiment, the image processing unit 105 includes a color conversion unit 600, and generates a post-color conversion CMOS image 602 from a pre-color conversion CMOS image 601, as shown in FIG. Then, the generated color component image of the post-color conversion CMOS image 602 is used as the CMOS color component image 201 in FIGS.

The color conversion applied by the color conversion unit 600 is performed as follows when machine learning is used, for example.
First, the CMOS image group for learning (resolution 6000×4000) is reduced to match the resolution of the SPAD image (1800×1200), and a small size (for example, 128×128) image patch group for learning is extracted therefrom. . Also, a teacher image patch group is extracted from the learning SPAD image group (resolution 1800×1200). At this time, patch extraction is performed so that cluster flaws of the SPAD image are not included in the teacher image patch. A color conversion model can be learned by deep learning or the like using such a group of image patches for learning and a group of teacher image patches.

It is also possible to use a Macbeth chart or the like as a color sample, use the colors on the CMOS side as inputs and the colors on the SPAD side as outputs, and calculate a color conversion matrix by the method of least squares or the like.

Synthesis of the color-converted CMOS image 602 and the SPAD image can be performed as described in the first or second embodiment, so description thereof is omitted here.

As described above, according to the third embodiment, by performing color conversion that brings the color components of the CMOS image closer to the color components of the SPAD image, color reproduction in combining the CMOS color component image and the SPAD color component image is improved. It can be improved further.

<Other embodiments>
Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device executes the program. It can also be realized by a process of reading and executing. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: imaging device, 101: control unit, 102: ROM, 103: RAM, 104-1, 104-2: imaging unit, 105: image processing unit, 106: recording unit, 107: display unit, 220, 222 to 224 243, 244, 542: same layer color component synthesizing unit 261 to 263: edge detecting unit 272 to 274: enlargement unit 281 to 283: different layer color component synthesizing unit , 501: resolution determination unit, 502: super-resolution unit

Claims (25)

a first image having a first resolution and a first color reproducibility obtained by imaging the same subject with a plurality of imaging means; a second resolution lower than the first resolution; acquisition means for acquiring a second image having a second color reproducibility higher than the first color reproducibility;
Synthesizing means for synthesizing the first image and the second image at a synthesis ratio according to the brightness and frequency components obtained from the first image and the second image to generate a synthesized image. and
When the brightness is the first brightness, a synthesis ratio of the color components of the second image is set higher than when the brightness is the second brightness, which is brighter than the first brightness. Image processing device. The synthesizing means is
first layering means for extracting color components of the first image and generating first layer images of a plurality of layers from the obtained first color component image;
second layering means for extracting color components of the second image and generating a second layer image of at least one of the plurality of layers from the obtained second color component image; and
generating a third layer image of each layer of the plurality of layers from the first layer image and the second layer image;
synthesizing the generated third layer images of the plurality of layers at a synthesis ratio of each layer according to the frequency component to generate a color component of the synthesized image;
When the amount of frequency components higher than a predetermined frequency among the frequency components is the first amount, the hierarchy is higher than the case where the amount is the second amount less than the first amount. 2. The image processing apparatus according to claim 1, wherein a composition ratio of said third layer image of is increased. When the first layer image and the second layer image exist in the same layer, the third layer image is generated by synthesizing at a composition ratio according to the brightness, and the second layer image is generated in the same layer. 3. The image processing apparatus according to claim 2, wherein the first hierarchical image is used as it is when there is no hierarchical image. The first layering means generates the first layer images of the plurality of layers with different resolutions by reducing the first color component image step by step,
The second layering means reduces the second color component image to at least one resolution lower than the resolution of the second color component image among the resolutions of the plurality of layers. , to generate the second hierarchical image. The first layering means converts the first color component image into a plurality of images of different frequency bands to generate the first layer images of the plurality of layers,
The second layering means generates the second layer image of the plurality of layers by converting the second color component image into the plurality of different frequency bands. 4. The image processing device according to item 2 or 3. 6. The synthesizing unit according to any one of claims 2 to 5, wherein said synthesizing means synthesizes said first layer image and said second layer image of the same layer using one synthesis ratio according to said brightness. 1. The image processing apparatus according to claim 1. 3. The synthesizing means synthesizes the first layer image and the second layer image of the same layer using a synthesis ratio corresponding to the brightness obtained for each pixel. 6. The image processing device according to any one of 5. 8. The image processing apparatus according to any one of claims 2 to 7, wherein said synthesizing means detects an amount of edges included in said third layer image as said amount of frequency components. 9. The synthesizing unit determines that the brightness is darker as the difference between the first layer image and the second layer image of the same layer is greater. 2. The image processing device according to item 1. The synthesizing means further includes transforming means for transforming the color components of the first image so as to approximate the color components of the second image,
10. The image processing apparatus according to any one of claims 1 to 9, wherein said synthesizing means synthesizes said converted first image and said second image. 11. The image processing apparatus according to claim 10, wherein said conversion means performs said conversion based on a color conversion model learned by machine learning. The conversion means uses image patches extracted from the first image and the second image as images for learning, and among the images for learning, image patches extracted from the second image as teacher images. 12. The image processing apparatus according to claim 11, wherein learning is performed. 13. The image processing apparatus according to claim 12, wherein said learning image is extracted after reducing said first image to the same resolution as said second image. 14. The image processing apparatus according to claim 12, wherein the teacher image is extracted so as not to include defective pixels of a predetermined size or larger. The conversion means uses a Macbeth chart as a color sample, inputs the color components of the first image, outputs the color components of the second image, and uses a color conversion matrix calculated by the method of least squares or the like. 11. The image processing apparatus according to claim 10, wherein said conversion is performed. The plurality of imaging means includes a first imaging means for accumulating charges according to incident light to generate the first image, and a second imaging means for counting the number of incident photons to generate the second image. 16. The image processing apparatus according to any one of claims 1 to 15, further comprising second imaging means. The acquiring means acquires a plurality of the second images,
The synthesizing means is
a determining means for determining the number of the plurality of second images acquired by the acquiring means to be used for synthesis;
conversion means for synthesizing the number of the second images determined by the determination means out of the plurality of second images, and converting the second images into an image with a higher resolution;
17. The image processing apparatus according to any one of claims 1 to 16, wherein said synthesizing means synthesizes said second image whose resolution has been converted with said first image. The determination means determines that the higher the average saturation calculated from the first image or the second image, or the smaller the number of pixels in the subject area included in the first image, or 18. The image processing apparatus according to claim 17, wherein the number of said second images used for said composition is increased as the amount of motion of a subject increases. The synthesizing means is
first layering means for extracting color components of the first image and generating first layer images of a plurality of layers from the obtained first color component image;
second layering means for extracting color components of the second image and generating a second layer image of at least one of the plurality of layers from the obtained second color component image; and
19. The determination means determines the number of images to be combined with the second image based on a difference between the first layer image and the second layer image of the same layer. The image processing device according to . 17. The converting means selects and synthesizes a second image having fewer scratches of a predetermined size or larger overlapping an edge portion from among the plurality of second images. 20. The image processing device according to any one of 19. 21. The image processing apparatus according to claim 20, wherein said conversion means synthesizes the selected second image without using pixels in an area corresponding to a flaw of a predetermined size or larger. . An image processing device according to any one of claims 17 to 21;
a first imaging means for imaging the first image;
a second imaging means for imaging the second image;
the first image pickup means and the and control means for controlling the second imaging means. An acquisition means obtains a first image having a first resolution and a first color reproducibility obtained by imaging the same subject with a plurality of imaging means, and a second image having a first resolution lower than the first resolution. a capturing step of capturing a second image having a resolution and a second color rendition higher than the first color rendition;
a synthesizing step in which synthesizing means synthesizes the first image and the second image at a synthesis ratio according to the brightness and frequency components obtained from the first image and the second image; have
When the brightness is the first brightness, a synthesis ratio of the color components of the second image is set higher than when the brightness is the second brightness, which is brighter than the first brightness. Image processing method. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 21. A computer-readable storage medium storing the program according to claim 24.

Images