JP2020102839A - Image processing device and control method of the same - Google Patents

Abstract

To make it possible to generate a more suitable synthetic image.SOLUTION: The image processing device includes: image acquisition means for acquiring a visible image and an infrared image; brightness acquisition means for acquiring brightness of the visible image; determination means for determining a synthesis ratio of the infrared image and the visible image for generating a synthetic image in which the infrared image and the visible image are synthesized based on the brightness acquired by the brightness acquisition means; synthesizing means for synthesizing the infrared image and the visible image based on the synthesis ratio determined by the determining means; and correction means for correcting the synthesis ratio so that a change amount of the synthesis ratio does not exceed a threshold if a change amount in the synthesis ratio exceeds the threshold.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for synthesizing a visible image and an infrared image.

Network camera systems are widely used, for example, for the purpose of monitoring intruders or intruding objects into public buildings and places, stores such as banks and supermarkets, dams, bases, and restricted areas such as airfields. For example, image data captured by a network camera is stored in a server via a network, and the server analyzes whether or not an intruding object exists based on the stored image data. Further, in order to clearly capture an image even in a dark place, there is an image pickup device having not only a visible light source but also an infrared light source. In this case, the operation of switching between imaging with visible light and imaging with infrared light is performed according to a predetermined condition. For example, based on the brightness information of the imaged image data, if the brightness value is above a specific brightness value, the visible light source is used for imaging, and if it is below the specific brightness value, the infrared light source is used. Take an image. In general, a visible image captured by a visible light source is acquired as a color image, and an infrared image captured by an infrared light source is acquired as a monochrome image.

Furthermore, there is also a method of synthesizing a visible image and an infrared image to generate a synthetic image. For example, there is known a method of synthesizing a visible image and an infrared image to generate a synthetic image that is a pseudo color image (hereinafter, referred to as MIX image). Further, Patent Document 1 discloses a technique of improving the image quality of an image in a situation where fog is generated by synthesizing a visible image and an infrared image.

JP, 2017-157902, A

However, when the composition ratio of the visible image and the infrared image when the composite image is generated based on the brightness threshold is switched, the brightness value in the composite image output before and after the switching may change significantly. Therefore, when the brightness of the subject fluctuates around the brightness threshold value, the composition ratio may change and the brightness value in the composite image may change significantly even if the brightness change is small. As a result, when the recognition process is performed based on the composite image, the state of the subject is erroneously recognized due to such a change in the brightness value. Further, when the user views the combined image, such a change in the brightness value makes the user feel uncomfortable.

The present invention has been made in view of such a problem, and an object thereof is to provide a technique capable of generating a more suitable composite image.

In order to solve the above problems, the image processing device according to the present invention has the following configuration. That is, the image processing device
Image acquisition means for acquiring a visible image and an infrared image,
Brightness acquisition means for acquiring the brightness of the visible image,
Based on the brightness acquired by the brightness acquisition unit, a determination unit that determines a combination ratio of the infrared image and the visible image for generating a combined image that combines the infrared image and the visible image. ,
A synthesizing means for synthesizing the infrared image and the visible image based on the synthesizing ratio determined by the determining means,
When the change amount of the combination ratio exceeds a threshold value, a correction unit that corrects the combination ratio so that the change amount of the combination ratio does not exceed the threshold value,
Have.

According to the present invention, it is possible to provide a technique capable of generating a more suitable composite image.

It is a figure which shows the structure of the camera apparatus in 1st Embodiment. It is a figure explaining the installation form of a camera device. It is a figure which shows the relationship between a brightness value and a synthetic|combination ratio. It is a flowchart which shows the whole operation|movement of a camera apparatus. It is a detailed flowchart which shows the update process of a synthetic|combination ratio. It is a figure which shows the relationship of the threshold value of a brightness value with respect to time. It is a figure which shows an example of a brightness value change with respect to time. FIG. 8 is a diagram showing an example of a correction result of a change in luminance value shown in FIG. 7. 6 is a detailed flowchart showing a correction process of a combination ratio in the first embodiment. It is a figure which shows the other example of a brightness value change with respect to time. It is a figure which shows the example of the correction result of the brightness value change shown in FIG. 9 is a detailed flowchart showing a correction process of a combination ratio in the second embodiment.

Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. Note that the following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
As a first embodiment of the image processing apparatus according to the present invention, a composite image obtained by capturing the same subject in the visible light region and the infrared light region and synthesizing the visible image and the infrared image obtained by the image capturing Will be described below by taking a camera device for network distribution as an example. In the following description, it is assumed that a visible moving image and an infrared moving image are acquired at a predetermined frame rate, and an image pair of the visible frame image and the infrared frame image acquired at the same timing is combined and distributed. .. Further, in the following description, a form in which a pseudo color image (MIX image) is generated as a composite image will be described, but it is also applicable to a form in which a monochrome image is generated.

<Device configuration>
FIG. 1 is a diagram showing a configuration of a camera device according to the first embodiment. Further, FIG. 2 is a diagram illustrating an installation form of the camera device.

The camera device 100 is communicatively connected to a user interface (UI) unit 120 via a network 121. A visible image captured by the camera device 100, an infrared image, or a MIX image that is a composite image generated based on these images is sent to the UI unit 120 via the network 121.

The camera device 100 includes a zoom lens 101 for visible image capturing, an image sensor 102, an A/D conversion processing unit 103, a development processing unit 104, a zoom lens 111 for infrared imaging, an image sensor 112, and an A/D conversion processing unit. 113 and a development processing unit 114. The camera device 100 also includes an image composition unit 105, a distribution unit 106, a brightness sensor 107, a brightness determination unit 108, and a composition ratio calculation unit 109.

The camera device 100 is configured to be capable of pan/tilt control in addition to zoom control by the zoom lenses 101 and 111. The zoom control and the pan/tilt control may physically control the imaging range or the imaging direction by controlling a drive motor or the like, or the imaging range or the imaging direction can be logically controlled by image processing. It may be controlled. The UI unit 120 transmits a control command such as pan/tilt/zoom to the camera device 100 based on an instruction from the user. The camera device 100 drives the zoom lenses 101 and 111 and a drive motor of a frame (not shown) according to the received control command. For example, when the control command is a zoom instruction, the drive motors of the zoom lenses 101 and 111 are driven so that the shooting range (shooting angle of view) indicated by the zoom instruction is achieved.

The zoom lens 101 is composed of an optical lens that transmits a visible light region, and the image pickup element 102 is composed of an element having sensitivity in the visible light region. The zoom lens 111 is an optical lens that transmits the infrared light region, and the imaging element 112 is an element that is sensitive to the infrared light region. The image pickup elements 102 and 112 are elements that output an electric signal according to the intensity of received light, and a CCD sensor or a CMOS sensor can be used. In addition, here, it is assumed that imaging is sequentially performed at a predetermined frame rate.

The analog signals output from the image pickup devices 102 and 112 are converted into digital signals by the A/D conversion processing units 103 and 113, and RAW image data that is digital data is generated. The development processing units 104 and 114 develop the received RAW image data and generate image data. Here, it is assumed that the development processing unit 104 generates color image data and the development processing unit 114 generates monochrome image data. For example, it can be set in the range of 0.1 to 30 frames/second, but is not limited to this range.

The image synthesizing unit 105 synthesizes the visible image acquired from the developing processing unit 104 and the infrared image acquired from the developing processing unit 114 according to a synthesizing ratio determined by a method described below to generate a MIX image. The distribution unit 106 distributes the MIX image generated by the image synthesis unit 105 to the network 121. The distribution unit 106 may distribute a compressed image obtained by compressing the MIX image according to an instruction from the UI unit 120.

The brightness sensor 107 is a sensor that generates brightness information of visible light around the place where the camera device 100 is installed. Instead of installing the brightness sensor 107, the brightness information of the visible image may be generated based on the information obtained by the image sensor 102. The brightness determination unit 108 acquires the brightness information generated by the brightness sensor 107 (brightness acquisition), and calculates the brightness value to be used for determining the combination ratio. The combination ratio calculation unit 109 calculates the combination ratio based on the brightness values calculated by the brightness determination unit 108.

FIG. 3 is a diagram showing a relationship (characteristic graph) between the luminance value and the composition ratio. The horizontal axis represents the composition ratio in image composition (ratio of infrared images in the composite image), and the vertical axis represents the brightness value of the visible image. The higher the brightness value of the visible image is, the brighter the subject (shooting range) is. Therefore, image combination is performed with a high ratio of the visible image (low value of the combination ratio). Then, there is such a relationship that the composition ratio continuously changes according to the change of the brightness value. The combination ratio calculation unit 109 determines the value of the combination ratio for the brightness value determined by the brightness determination unit 108 based on the characteristic graph of FIG. The image synthesis unit 105 synthesizes the visible image and the infrared image based on the value of the determined synthesis ratio to generate a MIX image.

It should be noted that the composition ratio of the infrared image in the composite image at the brightness value at which the switching is performed is 0 (zero) so that there is a continuous change in switching from visible image distribution to composite image distribution (or vice versa). % (That is, the visible image is 100%). In addition, the composition ratio of the infrared image in the composite image at the brightness value at which the switching is performed is 100% (so that there is a continuous change in switching from the infrared image distribution to the composite image distribution (or vice versa). That is, the visible image may be 0%). Further, it is preferable to set the rate of change of the composition ratio at relatively low luminance (right side of FIG. 3) to be smaller than the rate of change of the composition ratio at relatively high luminance (left side of FIG. 3).

<Device operation>
FIG. 4 is a flowchart showing the overall operation of the camera device. Specifically, the operation of the camera device 100 until a visible image and an infrared image are acquired and a MIX image is distributed is shown. In the following description, the frame image of interest (a pair of visible image and infrared image) at a certain timing is described, but it is executed for each frame image captured by the image capturing elements 102 and 112.

In S101, the image composition unit 105 acquires image data with visible light, and in S102, acquires image data with infrared light. In S103, the brightness determination unit 108 acquires the data from the brightness sensor 107 and acquires the brightness value.

In S104, the image compositing unit 105 determines whether or not to distribute the composite image according to the type of the distribution image previously designated by the UI unit 120. When it is determined that the composite image is to be distributed, the process proceeds to S105, and when it is determined that the composite image is not distributed (that is, the infrared image or the visible image is distributed), the process proceeds to S110.

In S110, the image composition unit 105 determines whether or not to deliver the infrared image. If it is determined that the infrared image is to be distributed, the process proceeds to S111, and if it is determined that the infrared image is not distributed (that is, the visible image is distributed), the process proceeds to S112. In S111, the image composition unit 105 selects an infrared image and outputs it to the distribution unit 106. On the other hand, in S112, the image composition unit 105 selects a visible image and outputs it to the distribution unit 106.

In S105, the luminance determination unit 108 determines whether or not a temporal change in luminance value (luminance value fluctuation) has occurred. If the brightness value has changed, the process proceeds to S106, and if the brightness value has not changed, the process proceeds to S107. In S106, the composition ratio calculation unit 109 updates the composition ratio. Details of updating the composition ratio will be described later with reference to FIG. In S107, the image combining unit 105 combines the infrared image and the visible image according to the determined combining ratio to generate a MIX image that is a combined image. In S108, the image composition unit 105 selects the generated composite image and outputs it to the distribution unit 106.

In S109, the distribution unit 106 distributes the image selected from any of S108, S111, and S112 and received from the image composition unit 105 to the network 121.

FIG. 5 is a detailed flowchart showing the composition ratio updating process (S106). Specifically, it is an update (calculation) process of the combination ratio used for MIX image generation, which is executed by the combination ratio calculation unit 109. The update process may be configured to be executed in synchronization with the overall operation shown in FIG. 4, or may be configured to be executed independently of the overall operation in FIG.

In S201, the brightness determination unit 108 calculates a brightness value. In S202, the combination ratio calculation unit 109 acquires characteristic information indicating the relationship between the luminance value and the combination ratio (for example, the characteristic graph shown in FIG. 3). The characteristic information may be stored in advance in the storage unit in the camera device 100, or may be acquired from the UI unit 120 or another device.

In S203, the synthesis ratio calculation unit 109 determines the value of the synthesis ratio with respect to the brightness value based on the characteristic information read in S202. In S204, the composition ratio calculation unit 109 outputs the value of the composition ratio determined in S203 to the image composition unit 105.

In step S205, the brightness determination unit 108 determines whether the brightness information from the brightness sensor 107 has been updated. If there is an update, the process proceeds to S201 and the process is repeated. If there is no update, the process ends.

As described above, by continuously changing the composition ratio according to the brightness around the place where the camera device 100 is installed, the brightness value in the composite image changes significantly even when the brightness change is small. It is possible to prevent the operation. That is, it becomes possible to generate and output an appropriate MIX image.

By the way, when the luminance value changes abruptly, the composition ratio may also change abruptly. Therefore, the composition ratio with respect to the target frame image (target image pair) is further corrected according to the change rate of the luminance value at the timing when the target frame image is acquired. Specifically, when a change in the brightness value occurs, a threshold value is set for the change in the brightness value according to the time corresponding to the change in the brightness value to prevent a sudden change in the composition ratio.

FIG. 6 is a diagram showing the relationship of the threshold value of the brightness value with respect to time. The brightness determination unit 108 corrects the brightness value shown in FIG. 6 to a brightness value indicated by the threshold value when the brightness change is equal to or more than the threshold value.

FIG. 7 is a diagram showing an example of changes in luminance value with respect to time. Specifically, an example of a temporal transition of the brightness value detected by the brightness sensor 107 is plotted. The markers 200, 201, 202, 203, and 204 are plotted with the luminance values at five consecutive timings as circles. Among the five markers, the temporal change of the marker 202 is below the threshold value and below. In this case, the brightness determination unit 108 corrects the marker 202 according to the threshold tilt information.

FIG. 8 is a diagram showing an example of the correction result of the brightness value change shown in FIG. 7. The result of correcting the marker 202 in FIG. 7 is the marker 205 in FIG. Further, since the position of the marker 202 is corrected to the position of the marker 205, the temporal change of the marker 203 with respect to the corrected marker 202 exceeds the threshold value and falls below the threshold value. Therefore, the position of the marker 203 is also corrected to the position of the marker 206 in FIG.

FIG. 9 is a detailed flowchart showing the correction processing of the combination ratio in the first embodiment. Specifically, it is a process executed by the brightness determination unit 108 and the combination ratio calculation unit 109 to correct (calculate) the combination ratio according to the change in brightness. It is assumed that this correction process is executed at a predetermined time interval. Further, it may be configured to be executed in synchronization with the update processing shown in FIG. 5, or may be configured to be executed independently of the update processing of FIG.

In S301, the brightness determination unit 108 reads the previously read brightness value corresponding to the image pair preceding the target image pair. The previously read luminance value is recorded in the recording medium, and the luminance determination unit 108 reads the previous luminance value from the recording medium. In S302, the brightness determination unit 108 acquires the current brightness value from the brightness sensor 107 and calculates the difference from the previous brightness value read in S301. In S303, the brightness determination unit 108 calculates a time interval for acquiring the brightness value from the brightness sensor 107.

In S304, the brightness determination unit 108 calculates the value of the slope (rate of change), which is the change amount of the brightness value with respect to time, from the difference between the brightness values calculated in S302 and the value of the time interval calculated in S303. In S305, the composition ratio calculation unit 109 calculates the amount of change in the composition ratio with respect to the value of the slope calculated in S304, for example, based on the characteristic graph shown in FIG.

In S306, the composition ratio calculation unit 109 compares the amount of change in the composition ratio calculated in S305 with the characteristic graph of the composition ratio, and determines whether or not the amount of change exceeds the threshold defined by the characteristic graph. .. If the change amount exceeds the threshold value defined by the characteristic graph, the process proceeds to S307, and if the change amount does not exceed the threshold value defined by the characteristic graph, the process proceeds to S308.

In S307, it is determined that there is a sudden change in the composition ratio, and the composition ratio is corrected. That is, as shown in FIGS. 7 and 8, the correction is performed so that the gradual change in the composition ratio is caused. On the other hand, in S308, it is determined that the composition ratio is moderate, and the composition ratio according to the characteristic graph is calculated.

In step S<b>309, the brightness determination unit 108 determines whether the brightness information from the brightness sensor 107 has received an update notification. If there is an update notification, the process proceeds to step S<b>301 and the process is repeated.

By the above processing, it becomes possible to suppress the abrupt change of the composition ratio when the abrupt change of the luminance value occurs.

As described above, according to the first embodiment, the composition ratio of the composite image of the video imaged by the imaging device having two light sources of infrared light and visible light is controlled according to the brightness value. This makes it possible to output a composite image in which erroneous recognition of the state of the subject and a sense of discomfort of the user are reduced even when brightness variation occurs during shooting.

(Second embodiment)
In the second embodiment, another mode of the process of calculating the composition ratio for a sudden change in luminance will be described. The device configuration and the overall operation are the same as those in the first embodiment, and the description thereof will be omitted. Below, the part different from the first embodiment (FIGS. 7 to 9) will be mainly described.

<Device operation>
FIG. 10 is a diagram showing another example of changes in luminance value with respect to time. Specifically, an example of a temporal transition of the brightness value detected by the brightness sensor 107 is plotted. The markers 300, 301, 302, 303, 304 are plotted as the circles indicating the luminance values at five consecutive timings. Among the five markers, the temporal change of the marker 302 is below the threshold value and is below the threshold value. In particular, the brightness values of the markers 301 and 303 before and after the marker 302 are almost the same, and only the brightness value of the marker 302 drops sharply. In this case, the brightness determination unit 108 corrects the marker 302 according to the threshold tilt information.

FIG. 11 is a diagram showing an example of the correction result of the brightness value change shown in FIG. The result of correcting the marker 302 in FIG. 10 becomes the marker 305 in FIG. Here, the marker 302 is determined to be a rapid change only at that timing, and is corrected to the brightness value of the marker 305 according to the brightness values of the markers 301 and 303 before and after the marker 302.

FIG. 12 is a detailed flowchart showing the correction processing of the combination ratio in the second embodiment. Specifically, it is a process executed by the brightness determination unit 108 and the combination ratio calculation unit 109 to correct (calculate) the combination ratio according to the change in brightness.

In step S401, the brightness determination unit 108 reads (sequentially acquires) the brightness value corresponding to the target image pair that is the evaluation target and the brightness values corresponding to before and after the target image pair (preceding and subsequent image pairs). For example, when the evaluation target is the luminance value of the marker 302, the luminance values at the timings of the three points of the markers 301 to 303 are read.

In S402, the brightness determination unit 108 calculates the difference between the brightness value of the evaluation target read in S401 and each of the brightness values acquired at the timings before and after the brightness value. In S403, the brightness determination unit 108 calculates the time interval of the brightness value read in S401. For example, in the example shown in FIG. 10, this corresponds to the time interval at which the brightness values of the markers 301, 302, 303 are acquired.

In step S404, the brightness determination unit 108 calculates the amount of change (inclination, rate of change) of the brightness value before and after the brightness value to be evaluated from the brightness difference calculated in step S402 and the time interval calculated in step S403. This makes it possible to determine the change state of the brightness value of the intermediate marker 302 among the brightness values of the three points. In particular, it can be determined whether or not there is a sudden temporary change with respect to the surrounding of the marker 302 located in the middle.

In step S405, the brightness determination unit 108 determines whether the change rate (brightness change amount) of the intermediate brightness value is equal to or greater than the threshold value. In the example shown in FIG. 10, the first luminance change rate between the markers 301 and 302, the second luminance change rate between the markers 302 and 303, and the third luminance change rate between the markers 301 and 303. The luminance change rate of is calculated. Then, it is determined whether or not the luminance value of the marker 302 is equal to or more than the threshold value. For example, when the third luminance change rate is smaller than the first luminance change rate and the third luminance change rate is smaller than the second luminance change rate, the luminance of the marker 302 located in the middle is rapidly changed temporarily. If it is determined that it is less than the threshold value, the process proceeds to S407.

In S406, the brightness determination unit 108 corrects the brightness value located in the middle according to the threshold value. For example, the marker 302 in FIG. 10 is corrected to the position of the marker 305 in FIG. That is, the brightness value of the marker 302 is corrected such that both the first brightness change rate and the second brightness change rate are less than the threshold value. On the other hand, in S407, the brightness determination unit 108 does not correct the brightness value located in the middle. In S408, the combination ratio calculation unit 109 calculates a combination ratio according to the characteristic graph based on the brightness values determined in S406 and S407.

In step S409, the brightness determination unit 108 determines whether the brightness information from the brightness sensor 107 has received an update notification. If there is an update notification, the process proceeds to step S401 and the process is repeated.

By the above processing, it becomes possible to suppress the abrupt change of the composition ratio when the abrupt temporary change of the brightness value occurs.

As described above, according to the second embodiment, the composition ratio of the composite image of the video imaged by the imaging device having two light sources of infrared light and visible light is controlled according to the brightness value. This makes it possible to output a composite image in which erroneous recognition of the state of the subject and discomfort of the user are reduced even when a temporary luminance change occurs during shooting.

(Modification)
In the above embodiment, an example in which the composition ratio of the composite image is indirectly controlled according to the brightness value has been described, but the composition ratio may be directly controlled. For example, if the composition ratio changes abruptly, and if the change is temporary, then after the abrupt change occurs based on the data of the change in the composition ratio over time before the abrupt change occurs. Correct the composition ratio. Specifically, extrapolation data of the composite ratio after a sudden change is extrapolated from the data of the change in the composite ratio with time before the abrupt change occurs, and the composite ratio is corrected so as to approximate the predicted data. To do. Here, whether or not the change of the composition ratio is abrupt is determined by whether or not the amount of change of the composition ratio is a predetermined value (threshold value) or more. Whether or not the change in the composition ratio is temporary is determined by whether or not the state in which the amount of change in the composition ratio is equal to or greater than a predetermined value (threshold value) is less than the predetermined time. By the above processing, it becomes possible to suppress the abrupt change of the composition ratio when the abrupt temporary change of the brightness value occurs.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Reference numeral 100 camera device; 101 zoom lens; 102 image sensor; 103 A/D conversion processing unit; 104 development processing unit; 105 image composition unit; 106 distribution unit; 107 brightness sensor; 108 brightness determination unit; 109 composite ratio calculation unit; Zoom lens; 112 image sensor; 113 A/D conversion processing unit; 114 development processing unit; 120 user interface unit; 121 network

Claims (11)

Image acquisition means for acquiring a visible image and an infrared image,
Brightness acquisition means for acquiring the brightness of the visible image,
Based on the brightness acquired by the brightness acquisition unit, a determination unit that determines a combination ratio of the infrared image and the visible image for generating a combined image that combines the infrared image and the visible image. ,
A synthesizing means for synthesizing the infrared image and the visible image based on the synthesizing ratio determined by the determining means,
When the change amount of the combination ratio exceeds a threshold value, a correction unit that corrects the combination ratio so that the change amount of the combination ratio does not exceed the threshold value,
An image processing apparatus comprising: The correction unit corrects the brightness so that the change amount of the brightness does not exceed the threshold value when the change amount of the brightness exceeds the threshold value,
The determining unit determines the combination ratio based on the luminance corrected by the correcting unit,
The image processing apparatus according to claim 1, wherein: 3. The image processing apparatus according to claim 1, wherein the determining unit determines the combination ratio so that a variation amount at a relatively low luminance is smaller than a variation amount at a relatively high luminance. .. 4. The deciding means decides the synthesizing ratio such that the proportion of the infrared image in the synthesizing image becomes smaller as the brightness becomes higher. Image processing device. The brightness acquisition means sequentially acquires the first brightness, the second brightness, and the third brightness,
The determining means calculates a first brightness change amount between the first brightness and the second brightness, and a second brightness between the second brightness and the third brightness. Calculate the amount of change,
The image according to claim 1, wherein the correction unit corrects the second brightness based on the first brightness change amount and the second brightness change amount. Processing equipment. The determining means calculates a third luminance change amount between the first luminance and the third luminance,
The correction unit corrects the second brightness when the third brightness change amount is smaller than the first brightness change amount and smaller than the second brightness change amount. 5. The image processing device according to item 5. The image processing apparatus according to claim 1, further comprising an information generating unit that generates the brightness based on an output of a brightness sensor installed around the image processing apparatus. A control method of an image processing device for synthesizing a visible image and an infrared image, the image acquiring step of acquiring a visible image and an infrared image,
A brightness acquisition step of acquiring the brightness of the visible image,
A determination step of determining a combination ratio of the infrared image and the visible image for generating a combined image in which the infrared image and the visible image are combined, based on the brightness acquired in the brightness acquisition step; ,
A combining step of combining the infrared image and the visible image based on the combining ratio determined by the determining step,
When the change amount of the combination ratio exceeds a threshold value, a correction step of correcting the combination ratio so that the change amount of the combination ratio does not exceed the threshold value,
A method for controlling an image processing apparatus, comprising: In the correction step, when the change amount of the brightness exceeds a threshold value, the brightness is corrected so that the change amount of the brightness does not exceed the threshold value,
In the determining step, the combination ratio is determined based on the brightness corrected in the correcting step,
The control method of the image processing apparatus according to claim 8, wherein. 10. The image processing apparatus according to claim 8, wherein, in the determining step, the combination ratio is determined such that a variation amount at a relatively low luminance is smaller than a variation amount at a relatively high luminance. Control method. A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images