JP2024054762A - Image processing device, control method and program - Google Patents

Abstract

[Problem] To provide an image processing device, an imaging device, a control method, and a program for improving the visibility of an image of a subject in a visible light image. [Solution] In an image processing device for improving the visibility of an image of a subject by synthesizing a visible light image and an infrared light image for an image of each frame constituting a moving image, a derivation process executed by an image processing unit acquires a visible light image and an infrared light image for each frame of the moving image, acquires an amount of infrared light for each frame, determines a target region for executing an enhancement process for each frame of the moving image based on the visible light image and the infrared light image, sets a target value indicating the visibility of an image obtained as a result of the enhancement process for the determined target region, and derives a correction amount for each of the visible light image and the infrared light image for each frame of the moving image. The derivation of the correction amount is based on the target value set for a reference frame and the difference in the amount of infrared light between the reference frame and the subsequent frame. [Selected Figure] Figure 7

Description

The present invention relates to an image processing device, an imaging device, a control method, and a program, and in particular to an image processing technique that improves the visibility of a visible light image using an infrared light image.

There is an image correction technology that uses the depth of the subject and the intensity of the aerosols, obtained using a radar device, and the brightness of sunlight to sharpen visible light images of scenes where aerosols such as fog or smoke are present in the atmosphere (Patent Document 1).

JP 2016-019201 A

However, in visible light images captured of scenes such as those described above, depending on atmospheric conditions, the image of the subject may not appear as a signal value in the first place. In such cases, even if the appropriate correction processing technique of Patent Document 1 is used, it may not be possible to obtain suitable correction results.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, an imaging device, a control method, and a program that improve the visibility of a subject image in a visible light image.

In order to achieve the above-mentioned object, the image processing device of the present invention is an image processing device that performs enhancement processing for each frame image constituting a moving image, by synthesizing a visible light image and an infrared light image to improve the visibility of an image of a subject, and includes a first acquisition means for acquiring a visible light image and an infrared light image of a subject captured for each frame of the moving image, a second acquisition means for acquiring an amount of infrared light related to an image capture scene for each frame of the moving image, a determination means for determining a target area for performing enhancement processing for each frame of the moving image based on the visible light image and the infrared light image, and an enhancement image obtained as a result of the enhancement processing for the target area determined by the determination means. The system has a setting means for setting a target value indicating the visibility of the image of the subject in the image, and a derivation means for deriving a correction amount for each of the visible light image and infrared light image acquired for each frame of the video in the enhancement process for that frame, the setting means determines the target value based on the visible light image and infrared light image acquired for a reference frame of the video, and the derivation means derives a correction amount for each of the visible light image and infrared light image acquired for a subsequent frame of the video based on the target value set for the reference frame and the difference in the amount of infrared light between the reference frame and the subsequent frame.

With this configuration, the present invention makes it possible to improve the visibility of the subject image in the visible light image.

FIG. 1 is a block diagram illustrating a functional configuration of an image capturing apparatus according to an embodiment and a modification of the present invention. FIG. 1 is a diagram for explaining an image capturing unit 105 according to an embodiment and a modified example of the present invention; FIG. 1 is a block diagram illustrating a functional configuration of an image processing unit 107 involved in enhancement processing according to an embodiment and a modification of the present invention. FIG. 1 shows an example of a visible light image and an infrared light image to which the present invention is applied. FIG. 1 is a diagram illustrating an example of image block division according to an embodiment and a modification of the present invention; FIG. 10 is a diagram illustrating the relationship between the amount of AC components of an image and the amount of edge enhancement applied to the image, according to the embodiment and modified examples of the present invention; 1 is a flowchart illustrating a derivation process executed in the image processing unit 107 according to the first embodiment of the present invention. FIG. 13 is a diagram for explaining a method for determining a target value for each region according to a seventh modified example of the present invention FIG. 13 is a diagram for explaining a method for setting a stepwise changing target value according to a ninth modified example of the present invention. 11 is a flowchart illustrating a light emission control process executed by the image capture device 100 according to the second embodiment of the present invention. FIG. 1 shows another example of a visible light image and an infrared light image to which the present invention is applied.

[Embodiment 1]
Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

The embodiment described below is an example in which the present invention is applied to an imaging device capable of capturing visible light images and infrared light images for each frame of a video, as an example of an image processing device. However, the present invention is applicable to any device that is capable of correcting visible light images for frames of a video using infrared light images. In addition, in this embodiment, "enhancement processing" is described as processing that improves visibility by emphasizing edges of the image of a subject included in a visible light image.

<Functional configuration of the imaging device>
FIG. 1 is a block diagram showing the functional configuration of an image capturing apparatus 100 according to an embodiment of the present invention.

The control unit 101 is a control device, such as a CPU, that controls the entire imaging device 100. The control unit 101 reads out the operation programs of each block of the imaging device 100 from the ROM 102, expands them in the RAM 103, and executes them to control the operation of each block.

ROM 102 is a non-volatile memory that can be electrically erased and recorded. ROM 102 stores the operation programs of each block of imaging device 100 as well as parameters and the like required for the operation of each block. RAM 103 is a rewritable volatile memory. RAM 103 is used not only as an area for expanding the operation programs of each block, but also as a temporary storage area for data output during the operation of each block.

The optical system 104 is composed of a lens group including a zoom lens and a focus lens. The optical system 104 guides a light beam related to the imaging scene to the imaging unit 105 and forms an optical image of the subject on the imaging surface. The imaging unit 105 is, for example, an imaging element such as a CCD or CMOS sensor. The imaging unit 105 photoelectrically converts the optical image formed on the imaging surface by the optical system 104 and outputs an analog image signal to the A/D conversion unit 106. In this embodiment, the imaging element of the imaging unit 105 is configured to be configured by arranging pixels to which an infrared light IR filter is applied in addition to pixels in a Bayer array to which a primary color RGB filter is applied, as shown in FIG. 2. That is, the imaging unit 105 can output an image signal related to visible light and an image signal related to infrared light by imaging. Although details will be described later, the imaging device 100 of this embodiment is configured to be able to perform imaging by irradiating a subject with near-infrared light, which easily passes through materials and changes its reflection mode depending on the characteristics of the material. Therefore, IR pixels receive light reflected from the subject that is in the near-infrared wavelength range (approximately 0.7 to 2.5 μm) and convert it photoelectrically.

The A/D conversion unit 106 converts the analog image signal output by the imaging unit 105 into a digital image signal. The generated digital image signal is stored in, for example, the RAM 103. In this embodiment, the imaging unit 105 outputs an analog image signal in which each pixel indicates the intensity of visible light or infrared light as shown in FIG. 2, so the A/D conversion unit 106 separates and outputs a digital image signal relating to visible light and a digital image signal relating to infrared light by A/D conversion. Hereinafter, the digital image signal relating to visible light will be referred to as a visible light image, and the digital image signal relating to infrared light will be referred to as an infrared light image.

The image processing unit 107 performs various image processing operations on the image data, such as correction processing and development processing. As will be described in detail later, the image processing unit 107 of this embodiment performs enhancement processing on the visible light image and infrared light image stored in the RAM 103, in order to improve the visibility of the subject in areas in the visible light image where the visibility (gradation) of the subject is reduced due to haze.

The recording unit 108 is a storage device for recording various types of information, such as a removable memory card. The recording unit 108 records the image obtained as a result of image processing by the image processing unit 107 as a recorded image via the RAM 103. The recorded image is not limited to a still image, but also includes a moving image.

The display unit 109 is a display device such as an LCD. The display unit 109 is used to display image data stored in the RAM 103 and image data recorded in the recording unit 108, and to display a user interface for receiving operational input from the user.

Detailed Configuration of Image Processing Unit
Next, the enhancement process executed in the image processing unit 107 of this embodiment will be described with reference to the block diagram of Fig. 3, showing the functional configuration involved in the process. The enhancement process of this embodiment is performed to improve the reduced visibility of the subject in the visible light image caused by fog, mist, smoke, etc. (hereinafter, these factors will be collectively referred to simply as "haze") contained in the imaging scene.

As shown in FIG. 4(a), the haze may appear as a whitish image in a visible light image, making the subject unclear, but as shown in FIG. 4(b), the haze may not reduce the visibility of the subject in an infrared light image. This is due to the property of near-infrared light being easily transmitted through substances, and the near-infrared light reflected on the surface of the subject behind the haze may penetrate the water molecules contained in the haze and reach the image sensor. Therefore, even if the gradation of the subject is lost due to the haze in the visible light image, the gradation of the subject may be displayed in the infrared light image without being affected by the haze. Therefore, when performing the enhancement process, the image processing unit 107 acquires a visible light image and an infrared light image captured at the same time. In the imaging device 100 of this embodiment, since the image sensor of the imaging unit 105 has a structure to which a filter of the type shown in FIG. 2 is applied, the image processing unit 107 performs the enhancement process using the visible light image and the infrared light image related to the image signal captured by the imaging unit 105 at the same timing.

The first image acquisition unit 301 and the second image acquisition unit 302 acquire information necessary for the enhancement process from the RAM 103. The enhancement process includes at least the visible light image to which the process is applied and an infrared light image that can be referenced for the correction (enhancement) of the visible light image. Therefore, the first image acquisition unit 301 acquires a visible light image (Bayer image), and the second image acquisition unit 302 acquires an infrared light image that was mainly taken at the same time as the visible light image.

The enhancement processing function of the imaging device 100 of this embodiment is not limited to use when capturing still images, but is configured to be usable when capturing moving images as well. Figure 3 shows the functional configuration of the image processing unit 107 when enhancement processing is performed primarily on each frame of a captured moving image.

When enhancement processing is performed on each frame of a moving image, the state of the infrared light image may change depending on the change over time in the ambient light of the captured scene (more specifically, the change over time in the amount of infrared light). For this reason, if enhancement processing is performed based on a standard according to the ambient light in each frame, the effect of improving visibility due to the enhancement processing (hereinafter, sometimes referred to as the enhancement effect) may vary between frames. Therefore, in the image processing unit 107 of this embodiment, the light amount acquisition unit 303 acquires information on the amount of infrared light in the captured scene as information further necessary for enhancement processing. The light amount acquisition unit 303 will be described below as deriving the amount of infrared light in the captured scene based on, for example, the infrared light image acquired by the second image acquisition unit 302.

The area determination unit 304 determines an area of the visible light image to be subjected to enhancement processing (hereinafter referred to as the target area) based on the visible light image acquired by the first image acquisition unit 301 and the infrared light image acquired by the second image acquisition unit 302.

The target derivation unit 305 derives a target value for the result of the enhancement process (a value that defines a state in which the reduction in visibility due to haze has been reduced). The target value is a value that quantifies the effect of haze removal in a visible light image, and is referenced for controlling the processing of the enhancement processing unit 306, as will be described in detail later.

The enhancement processing unit 306 performs enhancement processing on target areas where reduced visibility due to haze in the visible light image is observed, thereby eliminating this reduction and improving the visibility of the subject. The enhancement processing of this embodiment is realized by emphasizing the edges of the image of the subject that would normally appear in the target area (the subject whose gradation has been reduced due to haze). More specifically, the enhancement processing improves visibility by performing edge enhancement processing that emphasizes the AC components of the image of the subject in the target area of the visible light image. Depending on the state of the visible light image, the enhancement processing changes to either use only the visible light image or further synthesize an infrared light image, so the enhancement processing unit 306 controls its operation as appropriate.

The development processing unit 307 performs development processing on the visible light image obtained as a result of processing by the enhancement processing unit 306, and generates the final image to be recorded (a frame of a moving image or a still image). The image generated by the development processing unit 307 is stored in, for example, the RAM 103, and then recorded in the recording unit 108.

Overview of Enhancement Processing
The enhancement process performed by the image processing unit 107 of this embodiment during video shooting will be outlined below. As described above, the imaging device 100 of this embodiment is configured to be capable of performing enhancement processes when video shooting is performed, and the image processing unit 107 performs various processes to reduce variations in the enhancement effect between frames of the video. Therefore, the first image acquisition unit 301 and the second image acquisition unit 302 of the image processing unit 107 acquire, for each frame of the video, a visible light image and an infrared light image acquired in relation to that frame.

<Determination of target area>
First, the determination of the target region for the enhancement process performed by the region determination unit 304 will be described. The region to be determined as the target region is a region in the visible light image where the visibility of the subject is reduced due to haze. Therefore, the region determination unit 304 identifies the region where the visibility is reduced due to haze based on the difference in gradation between the visible light image acquired by the first image acquisition unit 301 and the infrared light image acquired by the second image acquisition unit 302.

In this embodiment, the amount of gradation in each image is evaluated based on the edge strength of the subject. The edge strength of the subject is quantified (an evaluation value is obtained) by integrating the edge signal (AC component signal) obtained by applying a band-pass filter to the image over a specified region. The region determination unit 304 defines M×N blocks as shown in FIG. 5 for the visible light image and infrared light image, and derives an evaluation value E for each block as the integral value of the edge signal of each image. The evaluation value E is calculated by using the edge signal value e(i,j) at pixel position (i,j) in the block, as follows:
The gradation information of each image is obtained by the band-pass filter. The region determining unit 304 obtains, as the gradation information of the visible light image, evaluation value information indicating the edge strength of each block of the visible light image to which the band-pass filter has been applied. The region determining unit 304 also obtains, as the gradation information of the infrared light image, evaluation value information indicating the edge strength of each block of the infrared light image to which the band-pass filter has been applied. In other words, the gradation information (evaluation value) of each image indicates the DC component separated by the band-pass filter, that is, the amount of AC components of the image (the sum of edge signal values).

When the region determination unit 304 acquires the gradation information of each image, it determines the target region to be subjected to the enhancement process based on the gradation information. The decision as to whether or not to designate a region as a target region is made on a block-by-block basis that was used to configure the gradation information. In other words, the region determination unit 304 determines, from among the blocks contained in the visible light image, the blocks to be subjected to the enhancement process as the target region.

A block that can be expected to have a favorable enhancement effect is one that has few gradations in the visible light image, but many gradations in the block at the same position in the infrared light image. In other words, if the visible light image has few gradations, but the difference in gradations between the visible light image and the infrared light image is large, it is considered that the image (shape) of the subject is expressed in the infrared light image, and therefore visibility is expected to improve by performing enhancement processing. Therefore, the region determination unit 304 determines as the target region a block whose evaluation value (edge strength of the block) of the visible light image is lower than the first threshold value and whose difference in evaluation values between the visible light image and the infrared light image (difference in edge strength of blocks at the same position) is greater than the second threshold value.

<Whether or not infrared images are used>
However, the enhancement process does not necessarily have to use an infrared light image. For example, if some gradation remains in the target area of the visible light image and the saturation of the area is high, the enhancement process can be performed using only the visible light image to obtain the effect of removing haze (improving visibility).

In contrast, if there is no gradation remaining in the target area of the visible light image and the saturation of that area is low, it is difficult to expect an improvement in visibility using only the visible light image. In other words, if the target area of the visible light image does not contain any information for reproducing the shape or color of the subject, it is not possible to improve the visibility of the subject image even if enhancement processing is performed using only the visible light image. For this reason, when it is difficult to expect an improvement in visibility using only the visible light image, it is possible to achieve the effect of improving visibility by using an infrared light image for enhancement processing.

Alternatively, the infrared light reflected by the subject and transmitted to the imaging device 100 may be partially absorbed by the presence of haze in the optical path, and these characteristics are apparent in the infrared image. In other words, the characteristics of reflected infrared light due to the presence of haze can also be captured in the infrared image in the near-infrared wavelength range. Therefore, when gradation remains in the area in the infrared image where the presence of haze is identified, i.e., when the amount of AC components in the infrared image is large, the use of the infrared image is expected to improve the visibility of the subject.

In this embodiment, the determination of whether or not to use an infrared light image is performed by the target derivation unit 305. Therefore, the target derivation unit 305 determines whether or not to use an infrared light image in the enhancement process based on the target region and the gradation information of each image. Specifically, this determination may be made by adopting criteria in any of the following aspects:

In one aspect, the target derivation unit 305 determines not to use the infrared light image when the evaluation value of any block included in the target region in the gradation information of the visible light image is higher than a third threshold (< first threshold) and the saturation of the block is higher than the first saturation threshold. That is, the target derivation unit 305 determines to perform enhancement processing using only the visible light image when the criterion is met, and to perform enhancement processing using the visible light image and the infrared light image when the criterion is not met.

In another aspect, the target derivation unit 305 determines to use the infrared light image when the evaluation value of any block included in the target region in the gradation information of the visible light image is lower than the fourth threshold (< the third threshold) and the saturation of the block is lower than the second saturation threshold. That is, the target derivation unit 305 determines to perform enhancement processing using the visible light image and the infrared light image when the criterion is met, and to perform enhancement processing using only the visible light image when the criterion is not met.

In yet another aspect, the target derivation unit 305 determines to use the infrared light image when the evaluation value of any block included in the target region in the gradation information of the infrared light image is higher than a fifth threshold value. That is, the target derivation unit 305 determines to perform enhancement processing using the visible light image and the infrared light image when the criterion is met, and to perform enhancement processing using only the visible light image when the criterion is not met.

<Generation of an Interpolated Infrared Image and Derivation of the Infrared Amount of a Captured Scene>
As described above, in the image signal output from the imaging unit 105 of this embodiment, IR pixels are included at a ratio of one in every 2×2 pixels as shown in Fig. 2. Therefore, the infrared light image acquired by the second image acquisition unit 302 does not include information on the pixel positions of R, G, and B. For this reason, the light amount acquisition unit 303 applies an interpolation process to the infrared light image acquired by the second image acquisition unit 302 to generate an interpolated infrared light image in which information on the pixels is supplemented. When generating the interpolated infrared light image, the light amount acquisition unit 303 may refer to the visible light image acquired by the first image acquisition unit 301.

The light amount acquisition unit 303 also derives the amount of infrared light in the captured scene of the corresponding frame based on the generated interpolated infrared light image. As described above, the amount of infrared light in the captured scene can change from moment to moment, and the signal strength of each pixel in the infrared light image changes accordingly. Therefore, when an infrared light image (interpolated infrared light image) is synthesized with a visible light image for each frame of a moving image to perform enhancement processing, the enhancement effect that appears in the resulting image can also vary. When the moving image output as a result of processing is played back, such variation in the enhancement effect can be perceived by the viewer as flickering, which can hinder a desirable viewing experience.

Therefore, in the image processing unit 107 of this embodiment, in order to ensure that the enhancement effect is uniform between frames, the infrared light amount acquired by the light amount acquisition unit 303 for each frame is used as a reference to control the processing content of the enhancement process to be different.

In this embodiment, the light amount acquisition unit 303 is described as deriving the amount of infrared light in the captured scene based on the interpolated infrared light image, but the implementation of the present invention is not limited to this. The amount of infrared light may be derived based on the infrared light image before interpolation, or based on the output of a sensor that measures ambient light and is provided separately from the imaging unit 105.

<Deriving target values>
In generating a moving image showing a homogeneous enhancement effect, the target derivation unit 305 derives a target value indicating the degree of haze removal in the image (visible light image) of each frame obtained as a result of the enhancement process. In this embodiment, the target derivation unit 305 derives the amount of AC components of the visible light image (hereinafter referred to as the output image) obtained as a result of the enhancement process as a target value. In the following description, the "amount of AC components of the output image" is defined as the sum of values obtained by multiplying the edge strength of the image to be processed by the edge enhancement amount (enhancement degree) of the image in the enhancement process. In other words, the enhancement process of this embodiment is assumed to obtain an enhancement effect in which haze is removed by emphasizing the edges of the image in the target region of the visible light image, and a numerical value that quantifies the edge strength after enhancement is used as the target value.

In order to make the enhancement effect uniform between frames, once a target value is set for a specific frame of a video, it is used commonly for the frames that follow that frame. In other words, the target value absolutely determines the guideline for the enhancement effect that appears in the output image, and once set, it is maintained until it becomes necessary to vary the enhancement effect. The specific frame from which the target value is derived (hereinafter referred to as the reference frame) may be, for example, the first frame (leading frame) captured in response to an operation input to start capturing a video to which enhancement processing has been applied. Alternatively, the reference frame may be the first frame after detection or after the angle of view has stabilized when a change in the captured scene is detected due to a change in the angle of view or panning of the imaging device 100.

Prior to deriving the target value, the target derivation unit 305 first determines the edge enhancement amount (correction amount) of each image used in the enhancement process related to the reference frame. More specifically, the target derivation unit 305 determines the edge enhancement amount E _vs of the visible light image of the reference frame based on the amount of AC components of the visible light image in the target region. The target derivation unit 305 also determines the edge enhancement amount E _vi of the interpolated infrared light image of the reference frame based on the difference between the amounts of AC components of the visible light image and the interpolated infrared light image in the target region. Here, in the enhancement process of this embodiment, one target value is set for the entire target region, so that the amount of AC components of the visible light image in the target region and the amount of AC components of the interpolated infrared light image are, for example, the average values of the amounts of AC components of blocks included in the target region.

The relationship between the amount of AC components of the visible light image and the edge enhancement amount _Ev of the visible light image is determined in advance as a characteristic shown in the graph indicated by the thick line in Fig. 6(a). When the amount of AC components of the visible light image is small, i.e., when the visible light image has few gradations, the visibility of the image of the subject in the visible light image is low. On the other hand, when the amount of AC components of the visible light image is large, i.e., when the visible light image has many gradations, the visibility of the image of the subject in the visible light image is high. For this reason, the smaller the amount of AC components of the visible light image, the larger the edge enhancement amount _Ev tends to be in order to strengthen the effect of haze removal, and the larger the amount of AC components, the smaller the edge enhancement amount _Ev tends to be in order to weaken the effect of haze removal.

On the other hand, the relationship between the difference in the amount of AC components between the visible light image and the interpolated infrared light image and the edge enhancement amount E _i of the interpolated infrared light image is determined in advance by the characteristic shown in the graph indicated by the thick line in Fig. 6 (b). As described above, when the infrared light image is used for enhancement processing, it is assumed that the amount of AC components of the visible light image is low. Therefore, when the difference in the amount of AC components between the visible light image and the interpolated infrared light image is small, the edge enhancement amount E _i is set to be large because it is expected that the visibility of the shape of the subject can be improved by amplifying the AC components of the interpolated infrared light image and strengthening the edge strength before combining. On the other hand, when the difference in the amount of AC components between the visible light image and the interpolated infrared light image is large, the edge strength of the interpolated infrared light image is originally strong, and it is expected that the visibility can be improved without amplifying the AC components of the interpolated infrared light image, so the edge enhancement amount E _i is set to be small.

In this way, when the edge enhancement amount E _vs of the visible light image and the edge enhancement amount E _is of the interpolated infrared light image for the reference frame are determined, the target derivation unit 305 derives the target value k. In this embodiment, the target value k is calculated by further using the edge strength AC _vs of the visible light image related to the reference frame (average value of blocks included in the target area) and the edge strength AC _is of the interpolated infrared light image (average value of blocks included in the target area) as follows:
That is, the target value k is derived as a value (sum of edge strengths) obtained by adding up the amounts of AC components after amplification of the visible light image and the interpolated infrared light image, which are combined in the enhancement process.

If it is determined that the infrared light image will not be used in the enhancement process of the target region, the edge enhancement amount E _is of the interpolated infrared light image is set to 0. In other words, since the interpolated infrared light image is not synthesized in the first place, there is no need for enhancement, and therefore the amount is set to 0.

Furthermore, the target value k is originally derived by the above-mentioned calculation formula, but in the following explanation, in order to facilitate understanding of the invention, when it is determined that an infrared light image is to be used in the enhancement processing of the target region, the edge enhancement amount E _vs of the visible light image is set to 0. This is because, when an infrared light image is used in the enhancement processing of the target region, the amount of AC components in the visible light image is small to begin with, and therefore no effect of edge enhancement is to be expected, and it is preferable to perform edge enhancement by giving priority to the AC components of the interpolated infrared light image.

That is, in the aspect described below, when an infrared light image is not used in the enhancement process, the AC components of the target region of the output image are determined by edge enhancement of only the visible light image, and the target value k is derived only from the edge enhancement amount E _vs and edge strength AC _vs of the visible light image. When an infrared light image is used in the enhancement process, the AC components of the target region of the output image are determined by edge enhancement of only the interpolated infrared light image, and the target value k is derived only from the edge enhancement amount E _is and edge strength AC _is of the infrared light image.

<Edge Enhancement Amount in Enhancement Processing>
In the enhancement processing performed by the enhancement processing unit 306, the output image is basically generated by amplifying the AC components of the target region of the image of each frame based on the amount of edge enhancement and adding the amplified AC components to the DC components of the visible light image of that frame.

More specifically, when an infrared light image is not used in the enhancement process, the enhancement processing unit 306 generates an AC component image for synthesis by amplifying only the pixel values of the target region for the AC components of the visible light image by the edge enhancement amount E _v . Then, the enhancement processing unit 306 generates an output image by adding the AC component image for synthesis to the DC component of the visible light image.

Furthermore, when an infrared light image is used for the enhancement process, the enhancement processing unit 306 generates an AC component image for synthesis by amplifying only the pixel values of the target region for the AC components of the interpolated infrared light image by the edge enhancement amount E _i , and then generates an output image by adding the AC component image for synthesis to the DC component of the visible light image.

At this time, the edge enhancement amount used to generate the AC component image for synthesis is determined based on the target value k determined for the reference frame. In the case of the reference frame, the edge enhancement amount (E _vs or E _is ) used to derive the target value k can be used as is. On the other hand, for frames after the reference frame, the amount of infrared light and the state of haze in the imaging scene may change over time. For this reason, the edge enhancement amount used for frames after the reference frame is derived using the edge strength (amount of AC components) of each image related to the frame and the target value k related to the reference frame as follows.

When the infrared light image is not used in the enhancement process, the edge enhancement amount E _v of the visible light image is calculated by using the edge strength AC _v of the visible light image of the corresponding frame and the target value k determined for the reference frame.
On the other hand, when an infrared light image is used for the enhancement process, the edge enhancement amount E _i of the interpolated infrared light image is calculated by using the edge strength AC _i of the interpolated infrared light image of the corresponding frame and the target value k determined for the reference frame as follows:
Here, ΔI is the difference between the amount of infrared light in the reference frame and the amount of infrared light in the corresponding frame.

In this way, the output images obtained as a result of the enhancement process in the reference frame and the subsequent frames can be controlled so that the edge strength of the subject in the target area is equal while achieving a haze removal effect. Therefore, the generated moving image has a consistent effect of improving the visibility of the subject image in the target area from frame to frame.

Derivation Processing
Hereinafter, a specific process of deriving an edge enhancement amount of the enhancement process performed on each frame when the image processing unit 107 of this embodiment generates a moving image in which the decrease in visibility due to haze is reduced will be described with reference to the flowchart in Fig. 7. The control unit 101 can cause the image processing unit 107 to execute the process corresponding to the flowchart by, for example, reading out a corresponding processing program stored in the ROM 102, expanding it in the RAM 103, and executing it. This derivation process will be described as being started, for example, when an operation input related to the start of shooting a moving image in which the decrease in visibility due to haze is reduced is detected, and is executed each time shooting of each frame of the moving image is performed.

In S701, the first image acquisition unit 301 and the second image acquisition unit 302 acquire the visible light image and the infrared light image related to the current frame from the RAM 103.

In S702, the region determination unit 304 derives gradation information for each image based on the visible light image and infrared light image acquired in S701.

In S703, the region determination unit 304 determines the target region of the visible light image of the current frame. More specifically, the region determination unit 304 determines the target region of the visible light image of the current frame based on the gradation information derived in S702.

In S704, the target derivation unit 305 determines whether or not to use an infrared light image in the enhancement process related to the current frame. More specifically, the target derivation unit 305 determines whether or not to use an infrared light image in the enhancement process based on the gradation information of the visible light image and the infrared light image generated in S702. If the target derivation unit 305 determines that an infrared light image will be used in the enhancement process, it proceeds to S705, and if it determines that an infrared light image will not be used, it proceeds to S706.

In S705, the light amount acquisition unit 303 generates an interpolated infrared light image from the infrared light image acquired in S701, and also derives the amount of infrared light in the captured scene.

In S706, the target derivation unit 305 determines whether the current frame is a reference frame. In this embodiment, in order to make the invention easier to understand, the reference frame is the first frame of a moving image in which reduced visibility degradation due to haze is reduced, so the target derivation unit 305 determines whether the current frame is the first frame. If the target derivation unit 305 determines that the current frame is a reference frame, it proceeds to S707, and if it determines that the current frame is not a reference frame, it proceeds to S709.

In S707, the target derivation unit 305 determines the amount of edge enhancement for each image based on the edge strengths of the visible light image and the interpolated infrared light image of the current frame, and outputs the amount of edge enhancement to be used in the enhancement process for the current frame to the enhancement processing unit 306. At this time, the target derivation unit 305 determines the amount of edge enhancement for each image based on the result of the determination in S704 as to whether or not to use an infrared light image in the enhancement process for the current frame. In the above-mentioned simple aspect, the target derivation unit 305 sets the amount of edge enhancement E _i of the interpolated infrared light image to 0 when an infrared light image is not used, and sets the amount of edge enhancement E _v of the visible light image to 0 when an infrared light image is used.

In S708, the target derivation unit 305 derives the target value k for the reference frame and completes this derivation process for the current frame. More specifically, the target derivation unit 305 derives the target value k based on the sum of the edge strengths of each image multiplied by the edge enhancement amount derived in S707. Information on the derived target value k for the reference frame is stored in RAM 103.

On the other hand, if it is determined in S706 that the current frame is not the reference frame, the target derivation unit 305 acquires the target value k for the reference frame as the target value for the enhancement process for the current frame in S709. That is, the target derivation unit 305 acquires the target value k derived for the reference frame in S708 from the RAM 103.

In S710, the target derivation unit 305 derives the edge enhancement amount for each image based on the edge strength of the visible light image and the interpolated infrared light image of the current frame, based on the target value acquired in S709. The target derivation unit 305 outputs the derived edge enhancement amount for each image to the enhancement processing unit 306 as the edge enhancement amount to be used in the enhancement processing for the current frame, and this derivation processing is completed.

Note that the atmospheric conditions during an image capture scene change over time, and the degree of visibility degradation due to haze may also change. In other words, during capture of a moving image in which visibility degradation due to haze is reduced, whether or not to use an infrared light image in the enhancement process for each frame may change depending on the degree of influence of the haze in the visible light image. For example, when the haze becomes gradually deeper, the edge strength, which is the gradation information of the visible light image, becomes smaller, so that even if only the visible light image is edge-enhanced, it may not be possible to improve visibility in the output image, and it becomes necessary to use an infrared light image. In such a case, the target value k derived for the reference frame based only on the edge strength and edge enhancement amount of the visible light image can be used to derive the edge enhancement amount of the interpolated infrared light image for the frame in which the haze has deepened. In other words, since the target value k derived for the reference frame is used for the subsequent frame, the effect of improving the visibility of the subject in the output image can be maintained even when not only the amount of infrared light in the image capture scene changes but also the state of the haze changes.

As described above, the image processing device of this embodiment can improve the visibility of the subject image in the visible light image while ensuring uniform visibility between frames of a moving image.

[Modification 1]
In the above embodiment, the target region is determined based on the gradation information of the visible light image and the infrared light image, but the present invention is not limited to this. In determining the target region, in addition to the gradation information, a transmission map indicating the transmittance of the haze in the captured scene may be used as auxiliary information. More specifically, the region determination unit 304 may determine, as the target region, a block that can be identified as hazy in the transmission map (where the transmittance is below a predetermined threshold) among blocks in which the difference in the evaluation values of the visible light image and the infrared light image exceeds a second threshold.

Note that the transmittance of the transmission map may vary over time depending on the measurement method and the material of the subject, resulting in the target area not being determined appropriately, and the enhancement effect in the output image may vary. For this reason, rather than using the transmission map obtained for each frame as is, a transmission map with stabilized transmittance information may be generated and used, for example by weighted addition of the transmission maps obtained for a certain number of recent frames.

[Modification 2]
In the above-described embodiment, the target area is determined based on the gradation information of the visible light image and the infrared light image, but the present invention is not limited to this. The target area may be specified by the user while viewing the through display on the display unit 109 during video shooting, for example. If the display unit 109 is a touch panel display, the specification may be performed in response to a touch operation on the display area, and in this case, the block including the coordinates where the touch operation was performed may be determined as the target area.

[Modification 3]
In the above embodiment, the amount of edge enhancement of the visible light image and the infrared light image related to the reference frame is determined based on the gradation information (amount of AC components) of these images and the characteristics shown in Fig. 6, but the embodiment of the present invention is not limited to this. The amount of edge enhancement of each image related to the reference frame may be determined based on a user-set value that specifies the strength of haze removal, for example.

[Modification 4]
In the above embodiment, information on the amount of infrared light is acquired for each frame and used to derive the amount of edge enhancement of the interpolated infrared light image for that frame, but the implementation of the present invention is not limited to this. The amount of infrared light in an image capture scene may fluctuate slightly up and down or sharply depending on factors such as the derivation method and other external factors. Therefore, if the amount of edge enhancement of the interpolated infrared light image is derived using the amount of infrared light acquired for each frame, there is a possibility that the effect of reducing the visibility reduction due to haze in the output image (enhancement effect) may vary.

In order to reduce such variation in the enhancement effect, the amount of infrared light pertaining to a frame may be derived as the amount of edge enhancement of the interpolated infrared light image by using the amount of infrared light pertaining to past frames, such as the average or weighted sum of the amounts of infrared light pertaining to the most recent frames. Alternatively, if the amount of infrared light pertaining to the current frame is suspected to be an error value, for example because the difference between the amount of infrared light pertaining to the current frame and the amount of infrared light pertaining to the most recent frame exceeds a threshold, the amount of infrared light used in processing may be determined only from the amount of infrared light pertaining to past frames. In this way, the variation in the enhancement effect can be further reduced.

[Modification 5]
In the above-described embodiment, when gradation is lost in the target area of the visible light image of a frame, the visibility of the subject image can be improved by synthesizing the AC components of the interpolated infrared light image of the same frame. On the other hand, when gradation is also lost in the target area of the interpolated infrared light image of the frame, a suitable enhancement effect may not be obtained even if the interpolated infrared light image is used.

In such a case, for example, an image obtained by combining an interpolated infrared light image relating to a past frame with an interpolated infrared light image relating to the current frame may be used in the enhancement process. Alternatively, instead of the interpolated infrared light image relating to the current frame, an interpolated infrared light image relating to a past frame may be used in the enhancement process relating to the current frame.

[Modification 6]
In the above-described embodiment, the edge enhancement amount of each image used in the enhancement process when capturing a moving image that reduces the visibility degradation due to haze is obtained based on the target value k derived for the reference frame, but the use of the enhancement processing unit 306 is not limited to this. For example, in a mode that enables capturing a still image while capturing a moving image to which the enhancement process is applied, the enhancement processing unit 306 may also be used for the enhancement process for the still image. In this case, for the still image, a capture instruction is accepted at a timing desired by the user while grasping the haze state in the capture scene, for example, so that the state of each image may be different from that of the reference frame for the moving image.

Therefore, for the enhancement process performed to generate the still image to be recorded, it is sufficient to derive and use the amount of edge enhancement for each image that is optimal for the relevant frame, without using the target value k for the reference frame. In other words, when video and still images are captured simultaneously, the amount of edge enhancement derived based on the target value obtained in the reference frame is used for the video, while the amount of edge enhancement derived independently is used for the still image. In this way, it is possible to record video images with a stable enhancement effect between frames, while recording still images with an enhancement effect optimized for the visible light image and infrared light image obtained at the time of capture.

[Modification 7]
In the above-described embodiment, all blocks in which the gradation information of the visible light image and the infrared light image satisfies a predetermined condition for the entire angle of view are defined as one target region, and one target value is set for the target region (or the entire angle of view). However, since the degree of visibility of subjects distributed within the angle of view is not necessarily uniform, even if enhancement processing is performed using the edge enhancement amount of each image derived for one target region defined for the entire angle of view, a suitable enhancement effect may not be obtained in some regions.

For this reason, for example, for the visible light image and the infrared light image shown in FIG. 4, multiple regions as shown in FIG. 8A may be defined as target regions based on the gradation information of these images. In the example of FIG. 8A, six regions EA ₁ to EA ₆ are shown in the entire angle of view. Here, the multiple regions are all composed of blocks that satisfy the condition that the evaluation value of the visible light image is lower than a first threshold value and the difference in the evaluation values of the visible light image and the infrared light image is greater than a second threshold value, as in the first embodiment. Each region is defined by grouping adjacent blocks among the blocks that satisfy the condition, in which at least one of the similarity of the evaluation value of the visible light image and the difference in the evaluation value of the visible light image and the infrared light image exceeds a predetermined value (high similarity of the evaluation values). By grouping in this way, blocks that are thought to show an image of the same subject can be defined as one region.

A method for determining a target value for each of the multiple regions will be described below using region _EA1 in Fig. 8(a) as an example. Assume that the evaluation values (amount of AC components) for each block in region _EA1 are distributed in the visible light image and the interpolated infrared light image as shown in Fig. 8(b). That is, blocks with evaluation values _V1 , _V2 , and _V3 are distributed in region _EA1 of the visible light image, and blocks with evaluation values _I1 and _I2 are distributed in region _EA1 of the interpolated infrared light image.

At this time, the target derivation unit 305 first obtains the edge enhancement amount for each block based on the gradation information (evaluation value) of each image, and derives a target value. For example, assume that two target values _k1 and _k2 (> _k1 ) are derived for the evaluation values of each block of the area _EA1 shown in Fig. 8(b) as shown in Fig. 8(c). The target derivation unit 305 determines the smallest target value ( _k1 ) among the blocks in the area _EA1 as the target value for that area, and uses it to derive the edge enhancement amount for that area in the subsequent frame.

In this modified example, the smallest target value among the blocks in the region is adopted, but the implementation of the present invention is not limited to this, and the target value of the region may be determined based on any criteria, such as the maximum target value in the region or the average target value.

In addition, in this modified example, the determination of one or more regions to be the target region is described as being performed by grouping based on the evaluation value, but the implementation of the present invention is not limited to this. For example, by referring to a depth map within the field of view (distance information of the subject in each block) as auxiliary information, blocks in which the image of the same subject appears can be grouped more accurately as the same region.

[Modification 8]
In the above embodiment, when a target value is derived in a reference frame, enhancement processing is performed so that the output image shows the edge strength of the target value in the frames following the reference frame, but the present invention is not limited to this. For example, the target value may be controlled so as to show the effect of gradually varying the degree of edge enhancement in order to inform the effect of haze removal.

In this case, the target derivation unit 305 stores the target value k derived by the method of the first embodiment in the reference frame as the final target value k _max at which the effect of haze removal is maximized in the RAM 103. The final target value k _max is set as a target value for deriving the edge enhancement amount of each image when a predetermined number of frames have passed from the reference frame. On the other hand, from the reference frame to the predetermined frame, the target derivation unit 305 determines the target value by increasing it stepwise from 0 as shown in FIG. 9. In the example shown in the figure, the target value from the reference frame to the (N-1)th frame is linearly determined so that the final target value k _max is reached at the Nth frame with the 0th frame being the reference frame. That is, from the reference frame to the (N-1)th frame, the target value is a variable value that gradually reaches the final target value k _max according to the number of frames passed from the reference frame.

Therefore, when the target derivation unit 305 determines that the current frame is the reference frame in S706 of the derivation process, it determines the final target value _kmax , and derives the target value k based on the final target value _kmax according to the number of frames elapsed from the reference frame.
and uses it to derive the edge enhancement amount for each image. Furthermore, when the number of elapsed frames m is equal to or greater than N, the target derivation unit 305 uses the final target value k _max as the target value k to derive the edge enhancement amount for each image. In this way, it is possible to present an appearance in which the haze is gradually removed and the visibility of the subject image is improved (the gradation is increased) during playback of the moving image.

In addition, the manner in which the target value is controlled to gradually show the effect of haze removal is not limited to when the application of the enhancement process is started during video capture, but may be when the application of the enhancement process is ended. In this case, contrary to the above-mentioned time of the start of application, the target value may be controlled to gradually attenuate from the final target value k _max to 0 according to the number of frames that have elapsed since the operation input related to the end of application was made. In this way, it is possible to present the appearance in which the haze gradually increases and the visibility of the subject image returns to the original visible light image state (the gradation decreases) during playback of the video.

[Embodiment 2]
In the above-described embodiment and modified example, in the enhancement process for a frame subsequent to the reference frame, the edge enhancement amount of each image is derived so as to be the target value determined for the reference frame. On the other hand, if the edge enhancement amount is too large, noise in the image is also emphasized, so that an output image with suitable image quality may not be obtained even if an interpolated infrared light image is synthesized. In other words, the edge enhancement amount of each image derived in the derivation process of embodiment 1 is derived based on the edge intensity of each image so that the output image will be the target value k determined for the reference frame, and there is a possibility that the image quality of the output image may not be guaranteed.

In other words, in a mode in which an infrared light image is used for enhancement processing, the shortfall in the edge strength of the target area of the interpolated infrared light image from the target value k, which is the edge strength of the output image, is compensated for by increasing the value of the edge enhancement amount by which the former is multiplied. For this reason, if the edge strength itself of the target area of the interpolated infrared light image is low, the edge enhancement amount will inevitably be large, which can lead to noticeable noise generation in the output image.

The amount of infrared light in an imaging scene can change due to changes in weather conditions, etc., and the edge strength (amount of AC components) of the target area of the interpolated infrared light image can also increase or decrease depending on the imaging scene. In other words, in imaging scenes where the amount of AC components of the target area of the interpolated infrared light image is small to begin with, a high value must be used as the amount of edge enhancement for the interpolated infrared light image, and there is a possibility that an output image of suitable image quality will not be obtained.

On the other hand, in order to ensure the image quality of the output image, it is also possible to set an upper limit on the amount of edge enhancement of the interpolated infrared light image. In this case, the target derivation unit 305 determines the amount of edge enhancement of the interpolated infrared light image to be the lower of either a value derived based on the target value k, the edge strength of the interpolated infrared light image, and the difference in the amount of infrared light, or a predetermined upper limit. However, in such an embodiment, there is a possibility that the edge strength of the output image obtained by the enhancement process will not reach the target value k.

In this embodiment, a method of dynamically controlling the gradation of the captured infrared light image so that an output image with a certain enhancement effect is recorded as a frame of a moving image while avoiding deterioration in the image quality of the output image will be described. More specifically, in the imaging device 100 of this embodiment, an upper limit value is set for the edge enhancement amount of each image, and the target derivation unit 305 derives a value equal to or less than the upper limit value as the edge enhancement amount for each frame. Then, when the amount of AC components of the interpolated infrared light image is small, the control unit 101 controls a light emitting device (not shown) provided in the imaging device 100 or connected to the imaging device 100 and controllable to irradiate infrared light. That is, when it is determined that the amount of AC components of the output image obtained as a result of the enhancement process of a certain frame is insufficient to the target value k, the light emission amount of the light emitting device is controlled so that infrared light that compensates for this is irradiated to the subject for capturing the infrared light image of the next frame.

<Light Emission Control Processing>
The light emission control process performed in the imaging device 100 of this embodiment will be described below with reference to the flowchart in Fig. 10. The process corresponding to this flowchart can be executed by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102 and loading it in the RAM 103. This light emission control process will be described as being started, for example, when the amount of edge enhancement for the interpolated infrared light image related to the current frame is derived in the derivation process, and the process is completed before the image signal related to the next frame is captured.

In S1001, the control unit 101 acquires from the RAM 103 information on the edge enhancement amount for each image related to the current frame derived in the derivation process, gradation information (edge strength) for each image of the current frame, and information on the target value k derived for the reference frame.

In S1002, the control unit 101 judges whether the amount of AC components of the output image reaches the target value k when the enhancement process for the current frame is performed using the information acquired in S1001. The judgment in this step may be performed by deriving the sum of the edge strengths of each image obtained in S1001 multiplied by the edge enhancement amount, as in the enhancement process, and comparing this sum with the target value k. Alternatively, the judgment in this step may be performed by comparing the amount of AC components of the output image generated by the enhancement processing unit 306 for the current frame with the target value k. If the control unit 101 judges that the amount of AC components of the output image reaches the target value k, it moves the process to S1003, and if it judges that it does not reach the target value k (it is insufficient), it moves the process to S1004.

In S1003, the control unit 101 acquires the amount of light emitted by the light emitting device when capturing the visible light image and the infrared light image of the current frame, and determines it as the amount of light emitted EM _a for the next frame. That is, since it is guaranteed that the amount of AC components of the output image will be the target value k in the amount of light emitted in the current frame, the control unit 101 reuses the amount of light emitted when capturing the current frame so that the next frame will have the same amount of light emitted.

On the other hand, if it is determined in S1002 that the amount of AC components in the output image does not reach the target value k, the control unit 101 derives the amount of light emitted by the light emitting device when capturing the visible light image and the infrared light image of the next frame in S1004. First, when using the edge enhancement amount E _i of the interpolated infrared light image of the current frame, the control unit 101 calculates the edge strength AC _n required for the interpolated infrared light image to make the amount of AC components in the output image reach the target value k.
Here, ΔI indicates the difference in the amount of infrared light between the captured scenes of the reference frame and the current frame. Furthermore, the control unit 101 calculates the difference ΔAC of the edge strength that is lacking in the interpolated infrared light image of the current frame based on the obtained edge strength _ACn .
Here, AC _i is the edge strength of the interpolated infrared light image of the current frame. Then, the control unit 101 uses the obtained ΔAC to calculate the light emission amount EM _a of the light emitting device when capturing the visible light image and the infrared light image of the next frame.
Here, EM indicates the amount of light emitted by the light emitting device when the current frame is captured.

In S1005, the control unit 101 controls the light emitting device to irradiate near-infrared light with the light emission amount EM _a of the light emitting device when capturing the derived visible light image and infrared light image of the next frame.

In this way, according to the image processing device of this embodiment, when the amount of AC components in the output image does not reach the target value k even when enhancement processing is performed using the interpolated infrared light image of the current frame, the interpolated infrared light image obtained in the next frame can be made into an appropriate state. In other words, by adjusting the amount of infrared light emitted to the subject in the next frame, the image processing device can increase the edge strength of the interpolated infrared light image obtained in the next frame and generate an output image that exhibits the same enhancement effect as the reference frame.

In the light emission control process of this embodiment, in order to determine whether or not additional infrared light irradiation is required in the next frame, it is determined in S1002 whether the amount of AC components of the output image reaches the target value k, but the implementation of the present invention is not limited to this. For example, in a mode in which an upper limit is set for the amount of edge enhancement, the determination in S1002 may be made by the control unit 101 determining whether the amount of edge enhancement derived for the interpolated infrared light image of the current frame is the upper limit. Alternatively, for example, in a mode in which an upper limit is not set for the amount of edge enhancement, the control unit 101 may determine whether the amount of edge enhancement derived for the interpolated infrared light image of the current frame exceeds a threshold value set as a level at which noise is not allowed.

In addition, in this embodiment, the light emission control process is described when the amount of AC components in the output image is insufficient to the target value k, but the implementation of the present invention is not limited to this. For example, in a mode in which a lower limit is set for the amount of edge enhancement of the interpolated infrared light image, if the interpolated infrared light image shows sufficient edge strength, performing enhancement processing may cause the amount of AC components in the output image to exceed the target value k. In other words, a mode in which the irradiation of the subject with infrared light by the light emitting device is sufficient can also be envisioned. In such a case, the control unit 101 can reduce the amount of light emitted by the light emitting device in the next frame compared to the current frame by performing processing similar to S1004 of the light emission control process described above.

[Modification 9]
In the above-mentioned second embodiment, the light emission amount of the light emitting device for the next frame is derived based on the amount of infrared light in the imaging scene, but the implementation of the present invention is not limited to this. For example, in an embodiment in which the imaging device 100 has a configuration for irradiating a subject with infrared light for other purposes such as AF, the operation of the light emitting device may be controlled so that the amount of infrared light emitted related to the irradiation is added to the amount of infrared light in the imaging scene and the amount of infrared light is subtracted from the amount of infrared light.

[Modification 10]
In the above-described second embodiment, an interpolated infrared light image in which the edge intensity of the target area is increased in the next frame is acquired by increasing the amount of light emitted by the light-emitting device in the next frame, but the implementation of the present invention is not limited to this. Even if the amount of light emitted by the light-emitting device provided in the imaging device 100 is increased, it is possible that the infrared light irradiated by the light-emitting device does not reach the subject, for example, when the subject is located far away.

For this reason, the control unit 101 may acquire information on the distance between the light emitting device and the subject, and if the distance exceeds a distance threshold indicating the range in which the light emitting device can be irradiated, may not control the light emitting device to increase the amount of light emitted. Note that if the amount of light emitted by the light emitting device is not increased because the distance between the light emitting device and the subject exceeds the distance threshold, the amount of edge enhancement may be set to a value greater than the upper limit, thereby prioritizing the maintenance of the enhancement effect.

[Modification 11]
In the light emission control process of the second embodiment described above, the amount of light emitted by the light emitting device is increased when the amount of AC components in the output image does not reach the target value k. However, there is an allowable limit to the amount of light emitted by the light emitting device. That is, if the amount of light emitted by the light emitting device in the current frame is already at the maximum amount, or if the amount of light emitted EM _a determined in S1004 exceeds the maximum amount, light emission control cannot be performed suitably. In this case, the amount of edge enhancement may be set to exceed the upper limit value, so that the maintenance of the enhancement effect is prioritized. Alternatively, an interpolated infrared light image with a suitable amount of AC components, which is acquired in a past frame under similar imaging conditions to the next frame, may be used for the enhancement process of the next frame.

[Modification 12]
In the above-described embodiment and modified examples, in order to facilitate understanding of the invention, it has been described that when an infrared light image is used in the enhancement process, the amount of edge enhancement _Ev of the visible light image is set to 0, and only the amount of edge enhancement _Ei of the interpolated infrared light image is derived. However, the implementation of the present invention is not limited to this, and in an aspect in which an infrared light image is used in the enhancement process, the amount of edge enhancement _Ev of the visible light image may be set to a significant value.

For example, the target derivation unit 305 sets the edge enhancement amount E _v of the visible light image based on a predetermined characteristic as shown in FIG. 6A. In this case, the edge enhancement amount E _i of the interpolated infrared light image is expressed as follows:
Here, AC _v is the edge strength of the visible light image of the current frame.

[Modification 13]
In the above-described embodiment and modified example, the enhancement process has been described as a process that clarifies the shape of the image of the subject by emphasizing the edges of the image of the subject whose visibility (gradation and saturation) has been reduced by haze, and gives an effect of removing the haze. However, the manner in which the visibility of the image of the subject is reduced in the visible light image is not limited to that caused by haze. For example, as shown in FIG. 11(a), even in an imaging scene in which the brightness of the environmental light such as a dark place is insufficient, the visibility of the image of the subject is reduced in the visible light image. On the other hand, in an infrared light image obtained by imaging the same subject, the image of the subject may appear with gradation remaining, as shown in FIG. 11(b). In this way, in an imaging scene in which the gradation of the same subject appears differently in the visible light image and the infrared light image, an output image in which the visibility of the image of the subject is improved can be generated by performing the enhancement process. In other words, the above-described derivation process can be applied to the imaging scene.

However, if the imaging scene is too dark or too bright, the visible light image is in a state in which it is difficult to distinguish the image of the subject, and information about the image of the subject may be lost in the DC component. Therefore, even if the AC component of the target area of the interpolated infrared light image is amplified according to the amount of edge enhancement and combined with the DC component of the visible light image in the enhancement process, a suitable output image cannot be obtained. In such cases, the visibility of the image of the subject in the output image can be improved by further combining the DC component of the interpolated infrared light image with the target area in the enhancement process.

Specifically, the enhancement processing unit 306 determines that the visible light image is too dark when the luminance of the target area of the visible light image falls below a first luminance threshold for determining low luminance and the BV value of the captured scene falls below a first BV threshold for determining low luminance. Alternatively, the enhancement processing unit 306 determines that the visible light image is too bright when the luminance of the target area of the visible light image exceeds a second luminance threshold for determining high luminance and the BV value of the captured scene exceeds a second BV threshold for determining high luminance. Then, in the enhancement processing when the enhancement processing unit 306 determines that the visible light image is too dark/too bright, it combines the AC components of the edge-emphasized visible light image and the interpolated infrared light image, and the DC component of the interpolated infrared light image with the DC component of the visible light image.

[Modification 14]
In the above-described embodiment and modified example, the IR filter applied to the imaging element of the imaging unit 105 is described as transmitting light in the wavelength range of near-infrared light, but the implementation of the present invention is not limited to this. The wavelength range of light transmitted by the IR filter is not limited to near-infrared light, and may include other wavelength ranges such as mid-infrared light and far-infrared light.

[Modification 15]
In the above-described embodiment and modified example, the imaging element of the imaging unit 105 is described as being configured to be capable of simultaneously acquiring a visible light image and an infrared light image, but the implementation of the present invention is not limited to this. The present invention is applicable to an aspect in which enhancement processing is performed using a visible light image and an infrared light image captured at the same time for the same subject, and it goes without saying that these images may be output from, for example, different imaging elements.

[Summary of the embodiment and modifications]
The disclosure of this specification includes the following image processing device, control method, and program.
(Item 1)
An image processing device that performs enhancement processing for improving visibility of an image of a subject by synthesizing a visible light image and an infrared light image for each frame image constituting a moving image, comprising:
a first acquisition means for acquiring the visible light image and the infrared light image obtained by capturing an image of the subject for each frame of the moving image;
A second acquisition means for acquiring an amount of infrared light related to an image capture scene for each frame of the moving image;
a determining means for determining a target region for performing the enhancement process for each frame of the video based on the visible light image and the infrared light image;
a setting means for setting a target value indicating visibility of the image of the subject in the image obtained as a result of the enhancement process for the target region determined by the determination means;
a derivation means for deriving a correction amount for each of the visible light image and the infrared light image acquired for each frame of the moving image in the enhancement process for that frame;
having
the setting means determines the target value based on the visible light image and the infrared light image acquired for a reference frame of the moving image;
the derivation means derives, for a frame subsequent to the reference frame of the moving image, the correction amount for each of the visible light image and the infrared light image acquired for the subsequent frame based on the target value set for the reference frame and the difference in the amount of infrared light between the reference frame and the subsequent frame.
(Item 2)
the enhancement processing is processing for emphasizing edges of the image of the subject,
2. The image processing device according to item 1, wherein the deriving means derives an amount of edge enhancement for each of the visible light image and the infrared light image as the amount of correction.
(Item 3)
3. The image processing device according to item 2, wherein the enhancement process is a process of amplifying AC components of the target region of the visible light image and the infrared light image based on the edge enhancement amount derived for each image, and further combining the AC components with a DC component of the visible light image.
(Item 4)
The image processing device described in item 3, characterized in that the derivation means derives the edge enhancement amount for each image so that a value obtained by amplifying and adding up AC components of the target regions of the visible light image and the infrared light image in the enhancement process for the subsequent frame becomes the target value.
(Item 5)
5. The image processing device according to any one of items 2 to 4, wherein the target value is an edge strength indicated by an AC component of an image obtained as a result of the enhancement process.
(Item 6)
The edge enhancement amount has an upper limit,
the image processing device further includes a control unit that controls a light emitting device that irradiates the subject with infrared light;
The image processing device described in item 5 is characterized in that, when the edge strength of the image obtained as a result of executing the enhancement processing on a first frame of the subsequent frames falls short of the target value, the control means increases the amount of infrared light emitted by the light-emitting device in a second frame following the first frame.
(Item 7)
a third acquisition means for acquiring a distance between the light emitting device and the subject,
7. The image processing device according to item 6, wherein the control means does not increase the amount of light emitted by the light emitting device when the distance to the subject included in the target area exceeds a distance threshold.
(Item 8)
The image processing device described in item 6 or 7, characterized in that when the amount of light emitted by the light emitting device in the first frame is at its maximum, the image processing device performs the enhancement process using the infrared light image acquired for a past frame that has imaging conditions similar to those of the second frame.
(Item 9)
The image processing device according to any one of items 1 to 8, wherein the second acquisition means acquires the amount of infrared light for each frame of the moving image based on the infrared light image acquired for that frame.
(Item 10)
The image processing device described in any one of items 1 to 8, characterized in that the second acquisition means acquires the amount of infrared light for each frame of the moving image based on the infrared light images acquired for a predetermined number of frames immediately preceding the frame in question.
(Item 11)
The image processing device described in any one of items 1 to 10, characterized in that the determination means determines the target area in each frame of the moving image based on gradation information of the visible light image and the infrared light image acquired relating to the frame.
(Item 12)
a determining unit that determines whether or not to combine the visible light image with the infrared light image in the enhancement process for each frame of the moving image based on gradation information of the visible light image or the infrared light image,
Item 12. The image processing device according to item 11, wherein the derivation means derives the correction amount of the visible light image for the subsequent frame not based on the amount of infrared light when the determination means determines not to combine the infrared light image with the visible light image.
(Item 13)
The image processing device described in any one of items 1 to 12, characterized in that the derivation means derives the correction amount for each of the visible light image and the infrared light image by using a variable value that gradually reaches the target value in accordance with the number of frames elapsed from the reference frame, instead of the target value set by the setting means, for a predetermined number of frames from the reference frame.
(Item 14)
The determining means determines a plurality of regions as the target regions for each frame of the moving image,
14. The image processing device according to any one of items 1 to 13, wherein the setting means sets the target value for each of the plurality of regions.
(Item 15)
The camera further includes an input unit for receiving an instruction to capture a still image while capturing an image of the subject for each frame of the moving image,
The image processing device described in any one of items 1 to 14, characterized in that when an instruction to capture a still image is accepted by the input means, the derivation means derives a correction amount for the still image in the enhancement processing for each of the visible light image and the infrared light image acquired relating to the frame for which the shooting instruction was accepted, without being based on the target value.
(Item 16)
The present invention further includes an imaging means for capturing the visible light image and the infrared light image,
16. The image processing device according to any one of items 1 to 15, wherein the first acquisition means acquires the visible light image and the infrared light image captured by the imaging means.
(Item 17)
17. The image processing device according to item 16, wherein the imaging means includes one imaging element in which pixels to which a filter that transmits visible light is applied and pixels to which a filter that transmits infrared light is applied are arranged.
(Item 18)
A method for controlling an image processing device that executes an enhancement process for improving visibility of an image of a subject by synthesizing a visible light image and an infrared light image for each frame image constituting a moving image, comprising the steps of:
a first acquisition step of acquiring the visible light image and the infrared light image obtained by capturing an image of the subject for each frame of the moving image;
a second acquisition step of acquiring an amount of infrared light related to an image capture scene for each frame of the moving image;
determining a target region for performing the enhancement process for each frame of the video based on the visible light image and the infrared light image;
a setting step of setting a target value indicating visibility of an image of the subject in the image obtained as a result of the enhancement process for the target region determined in the determination step;
a derivation step of deriving a correction amount for each of the visible light image and the infrared light image acquired for each frame in the enhancement process of the moving image;
having
In the setting step, the target value is determined based on the visible light image and the infrared light image acquired in relation to a reference frame of the moving image,
a control method characterized in that, in the derivation step, for a frame subsequent to the reference frame of the moving image, the correction amount for each of the visible light image and the infrared light image acquired for the subsequent frame is derived based on the target value set for the reference frame and the difference in the amount of infrared light between the reference frame and the subsequent frame.
(Item 19)
16. A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 15.
[Other embodiments]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: imaging device, 101: control unit, 105: imaging unit, 107: image processing unit, 301: first image acquisition unit, 302: second image acquisition unit, 303: light amount acquisition unit, 304: area determination unit, 305: target derivation unit, 306: enhancement processing unit

Claims (19)

An image processing device that performs enhancement processing for improving visibility of an image of a subject by synthesizing a visible light image and an infrared light image for each frame image constituting a moving image, comprising:
a first acquisition means for acquiring the visible light image and the infrared light image obtained by capturing an image of the subject for each frame of the moving image;
A second acquisition means for acquiring an amount of infrared light related to an image capture scene for each frame of the moving image;
a determining means for determining a target region for performing the enhancement process for each frame of the video based on the visible light image and the infrared light image;
a setting means for setting a target value indicating visibility of the image of the subject in the image obtained as a result of the enhancement process for the target region determined by the determination means;
a derivation means for deriving a correction amount for each of the visible light image and the infrared light image acquired for each frame of the moving image in the enhancement process for that frame;
having
the setting means determines the target value based on the visible light image and the infrared light image acquired for a reference frame of the moving image;
the derivation means derives, for a frame subsequent to the reference frame of the moving image, the correction amount for each of the visible light image and the infrared light image acquired for the subsequent frame based on the target value set for the reference frame and the difference in the amount of infrared light between the reference frame and the subsequent frame. the enhancement processing is processing for emphasizing edges of the image of the subject,
2 . The image processing apparatus according to claim 1 , wherein the deriving means derives an amount of edge enhancement for each of the visible light image and the infrared light image as the amount of correction. The image processing device according to claim 2, characterized in that the enhancement process amplifies the AC components of the target region of the visible light image and the infrared light image based on the edge enhancement amount derived for each image, and further combines them with the DC components of the visible light image. The image processing device according to claim 3, characterized in that the derivation means derives the edge enhancement amount for each image such that the sum of the amplified AC components of the target areas of the visible light image and the infrared light image in the enhancement process for the subsequent frame is the target value. The image processing device according to claim 2, characterized in that the target value is the edge strength indicated by the AC components of the image obtained as a result of the enhancement process. The edge enhancement amount has an upper limit,
the image processing device further includes a control unit that controls a light emitting device that irradiates the subject with infrared light;
The image processing device described in claim 5, characterized in that the control means increases the amount of infrared light emitted by the light-emitting device in a second frame following the first frame when the edge strength of the image obtained as a result of performing the enhancement processing on a first frame of the subsequent frames falls short of the target value. a third acquisition means for acquiring a distance between the light emitting device and the subject,
7. The image processing device according to claim 6, wherein the control means does not increase the amount of light emitted by the light emitting device when the distance to the subject included in the target area exceeds a distance threshold value. The image processing device according to claim 6, characterized in that, when the amount of light emitted by the light emitting device in the first frame is at its maximum, the image processing device executes the enhancement process using the infrared light image acquired in relation to a previous frame that has similar imaging conditions to those of the second frame. The image processing device according to claim 1, characterized in that the second acquisition means acquires the amount of infrared light for each frame of the moving image based on the infrared light image acquired for that frame. The image processing device according to claim 1, characterized in that the second acquisition means acquires the amount of infrared light for each frame of the moving image based on the infrared light images acquired for a predetermined number of frames immediately preceding the frame in question. The image processing device according to claim 1, characterized in that the determining means determines the target area in each frame of the moving image based on gradation information of the visible light image and the infrared light image acquired for that frame. a determining unit that determines whether or not to combine the infrared light image with the visible light image in the enhancement process for each frame of the moving image based on gradation information of the visible light image or the infrared light image,
The image processing device according to claim 11, characterized in that, when the judgment means judges not to combine the infrared light image with the visible light image, the derivation means derives the correction amount of the visible light image for the subsequent frame without being based on the amount of infrared light. The image processing device according to claim 1, characterized in that the derivation means derives the correction amount for each of the visible light image and the infrared light image by using a variable value that gradually reaches the target value according to the number of frames elapsed from the reference frame, instead of the target value set by the setting means, for a predetermined number of frames from the reference frame. The determining means determines a plurality of regions as the target regions for each frame of the moving image,
2. The image processing apparatus according to claim 1, wherein said setting means sets the target value for each of said plurality of regions. The camera further includes an input unit for receiving an instruction to capture a still image while capturing an image of the subject for each frame of the moving image,
The image processing device according to claim 1, characterized in that, when an instruction to capture a still image is accepted by the input means, the derivation means derives a correction amount for the still image in the enhancement processing for each of the visible light image and the infrared light image acquired relating to the frame for which the shooting instruction was accepted, without being based on the target value. The present invention further includes an imaging means for capturing the visible light image and the infrared light image,
The image processing apparatus according to claim 1 , wherein the first acquisition means acquires the visible light image and the infrared light image captured by the imaging means. The image processing device according to claim 16, characterized in that the imaging means includes one imaging element in which pixels to which a filter that transmits visible light is applied and pixels to which a filter that transmits infrared light is applied are arranged. A method for controlling an image processing device that executes an enhancement process for improving visibility of an image of a subject by synthesizing a visible light image and an infrared light image for each frame image constituting a moving image, comprising the steps of:
a first acquisition step of acquiring the visible light image and the infrared light image obtained by capturing an image of the subject for each frame of the moving image;
a second acquisition step of acquiring an amount of infrared light related to an image capture scene for each frame of the moving image;
determining a target region for performing the enhancement process for each frame of the video based on the visible light image and the infrared light image;
a setting step of setting a target value indicating visibility of an image of the subject in the image obtained as a result of the enhancement process for the target region determined in the determination step;
a derivation step of deriving a correction amount for each of the visible light image and the infrared light image acquired for each frame in the enhancement process of the moving image;
having
In the setting step, the target value is determined based on the visible light image and the infrared light image acquired in relation to a reference frame of the moving image,
a control method characterized in that, in the derivation step, for a frame subsequent to the reference frame of the moving image, the correction amount for each of the visible light image and the infrared light image acquired for the subsequent frame is derived based on the target value set for the reference frame and the difference in the amount of infrared light between the reference frame and the subsequent frame. A program for causing a computer to function as each of the means of an image processing device according to any one of claims 1 to 15.

Images