JP7484544B2 - Infrared image processing device, infrared imaging system, and infrared image processing method - Google Patents

Description

The present invention relates to an infrared image processing device, an infrared imaging system, and an infrared image processing method that correct the sensitivity of an infrared image.

Far-infrared cameras often use thermal detection elements that convert the temperature change caused by the absorption of infrared rays into an electrical signal. Thermal detection elements include bolometer, thermopile, and focal point types. Bolometers are elements that detect the temperature change caused by the absorption of infrared rays as a resistance change, and have a simple structure that is suitable for miniaturization. Microbolometers that use amorphous silicon (a-Si) or vanadium oxide (VOx) are widely used (see, for example, Patent Document 1).

The sensitivity of a microbolometer is highly dependent on the environmental temperature. In a typical far-infrared camera that uses a microbolometer, the gain corresponding to the measured environmental temperature is determined based on the sensitivity characteristics with respect to the environmental temperature, and the brightness signal is corrected. If the sensitivity for the entire temperature range is adjusted with one sensitivity curve, the range of the brightness signal may become too wide and may not fit within the dynamic range of the brightness signal. In particular, for automotive applications, a wide temperature range is required, and a support range of approximately -40°C to +80°C may be required.

One possible solution is to compress the dynamic range of the luminance signal by dividing one sensitivity curve into multiple ones, and then centering the multiple divided sensitivity curves so that the overall sensitivity range is narrowed.

JP 2012-15705 A

When using multiple sensitivity curves divided as described above, large differences in sensitivity occur at the points where the sensitivity curves switch. This difference in sensitivity can cause unnatural changes in the image when the sensitivity curves switch, to the extent that they cause discomfort to the user.

This embodiment was made in consideration of these circumstances, and its purpose is to provide a technology that suppresses degradation of image quality caused by adjusting the sensitivity of an infrared detection element to the environmental temperature.

In order to solve the above problem, an infrared image processing device according to a certain aspect of the present embodiment includes an image acquisition unit that acquires an infrared image from an infrared imaging device, a temperature acquisition unit that acquires an environmental temperature of an infrared detection element included in the infrared imaging device, and a sensitivity characteristic correction unit that corrects the sensitivity characteristic based on the sensitivity characteristic of the infrared detection element with respect to the environmental temperature and the acquired environmental temperature. The sensitivity characteristics are defined as a plurality of sensitivity characteristics divided into different temperature ranges so that a luminance signal of the infrared image is not saturated, and when a first sensitivity characteristic corresponds to a higher environmental temperature of two sensitivity characteristics adjacent to each other in a temperature range and a second sensitivity characteristic corresponds to a lower environmental temperature, the sensitivity characteristic correction unit corrects the sensitivity of the first sensitivity characteristic to match the sensitivity of the second sensitivity characteristic at the temperature at the switching point in a temperature range in which the sensitivity of the first sensitivity characteristic is lower than the sensitivity of the second sensitivity characteristic at the temperature at the switching point between the first sensitivity characteristic and the second sensitivity characteristic.

Another aspect of this embodiment is an infrared image processing method. This method includes the steps of acquiring an infrared image from an infrared imaging device, acquiring an environmental temperature of an infrared detection element included in the infrared imaging device, and correcting the sensitivity characteristic based on the sensitivity characteristic of the infrared detection element with respect to the environmental temperature and the acquired environmental temperature. The sensitivity characteristics are defined as a plurality of sensitivity characteristics divided into different temperature ranges so that the luminance signal of the infrared image is not saturated, and when a first sensitivity characteristic corresponds to a higher environmental temperature of two adjacent sensitivity characteristics in a temperature range and a second sensitivity characteristic corresponds to a lower environmental temperature, the correcting step corrects the sensitivity of the first sensitivity characteristic to match the sensitivity of the second sensitivity characteristic at the temperature at the switching point in a temperature range in which the sensitivity of the first sensitivity characteristic is lower than the sensitivity of the second sensitivity characteristic at a switching point temperature between the first sensitivity characteristic and the second sensitivity characteristic.

In addition, any combination of the above components, and conversions of the expressions of this embodiment between methods, devices, systems, recording media, computer programs, etc., are also valid aspects of this embodiment.

This embodiment makes it possible to suppress degradation of image quality caused by adjusting the sensitivity of the infrared detection element to the environmental temperature.

1 is a diagram showing a configuration of an infrared imaging system according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of a sensitivity curve divided into a plurality of parts. FIG. 13 is a diagram showing a plurality of sensitivity curves after correction of the first sensitivity curve. FIG. 13 is a diagram showing a plurality of sensitivity curves after correction of the first sensitivity curve and the second sensitivity curve. FIG. 13 shows a number of sensitivity curves after another correction of the first sensitivity curve.

Figure 1 is a diagram showing the configuration of an infrared imaging system 1 according to an embodiment of the present invention. The infrared imaging system 1 according to the embodiment includes an infrared imaging device 10 and an infrared image processing device 20. In this embodiment, the infrared imaging device 10 is assumed to be a far-infrared image imaging device that captures far-infrared rays of 8 um to 14 um. Unlike near-infrared rays, far-infrared rays can be imaged without providing a light source.

The infrared imaging device 10 includes a microbolometer 11, an amplifier 12, an AD converter 13, a thermistor 14, and an AD converter 15. In this embodiment, an example is assumed in which the microbolometer 11 is used as the far-infrared detection element.

The microbolometer 11 is a non-cooled thermal detection element. The microbolometer 11 includes a focal plane array (FPA) in which multiple bolometers are arranged in a two-dimensional lattice (e.g., 640 x 480, 1024 x 768). Each bolometer constituting one pixel has an absorption layer made of amorphous silicon or vanadium oxide. When far-infrared rays are incident on the absorption layer, the temperature of the absorption layer changes, and the resistance value of the absorption layer changes. By applying a predetermined voltage to two electrodes on the absorption layer, the change in the resistance value of the absorption layer is detected as a change in current value. The current value of each bolometer is converted into a voltage and output as a luminance signal of each pixel defined by an electrical signal.

The amplifier 12 amplifies the voltage output from each pixel with a predetermined gain. The AD converter 13 converts the analog voltage of each pixel amplified by the amplifier 12 into a digital value at a predetermined sampling period. In this embodiment, it converts into a 14-bit digital value.

The infrared imaging device 10 outputs an infrared thermal image, in which each pixel is defined by a 14-bit luminance signal, to the infrared image processing device 20. If it is desired to generate a thermal image with higher resolution, the analog voltage output from each pixel may be converted to a digital value of 16 bits or more. Conversely, if a low-cost configuration is desired, it may be converted to a digital value of 12 bits or less.

The thermistor 14 is a temperature detection element that is installed near the microbolometer 11 and is used to measure the environmental temperature around the microbolometer 11. The temperature around the microbolometer 11 is affected by the temperature of the lens (not shown) placed in front of the microbolometer, in addition to the temperature of the microbolometer itself.

The thermistor 14 is connected in series with a resistive element, and changes in the environmental temperature are detected as changes in the divided voltage. The AD converter 15 converts the analog voltage corresponding to the environmental temperature into a digital value. The infrared imaging device 10 outputs the measured environmental temperature to the infrared image processing device 20.

The thermistor 14 does not necessarily have to be installed inside the infrared imaging device 10, and may be installed outside the infrared imaging device 10 as long as it is located in a position that allows the ambient temperature of the microbolometer 11 to be estimated. The thermistor 14 is also an example of a temperature detection element, and other temperature detection elements such as a thermocouple may also be used.

The infrared image processing device 20 includes an image acquisition unit 21, a defective pixel correction unit 22, a non-uniformity correction unit 23, a sensitivity adjustment unit 24, an auto gain control unit 25, a temperature acquisition unit 26, and a sensitivity characteristic correction unit 27. These components can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As hardware resources, a CPU, ROM, RAM, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ISP (Image Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and other LSIs can be used. As software resources, programs such as firmware can be used.

The image acquisition unit 21 acquires an infrared image from the infrared imaging device 10. The defective pixel correction unit 22 corrects the luminance signal of a defective pixel in which an abnormality has occurred in the FPA, using interpolated data generated from normal pixels surrounding the defective pixel.

In a typical microbolometer 11, the gain and offset are non-uniform for each pixel. The non-uniformity correction unit 23 corrects the non-uniformity of the gain and offset between pixels. For example, with external incident light blocked, a blackbody shutter is used to uniformly irradiate the microbolometer 11 with far-infrared light, and a correction table is generated by deriving the gain and offset of each pixel so that the output of all pixels in the microbolometer 11 is constant. The non-uniformity correction unit 23 refers to the correction table to adjust the gain and offset of each pixel.

The sensitivity adjustment unit 24 adjusts the gain of the luminance signal of each pixel based on the sensitivity characteristics of the microbolometer 11 to the environmental temperature to adjust the sensitivity of the infrared image. The details of the sensitivity characteristics to the environmental temperature will be described later.

The auto gain control unit 25 converts the digital signal format of the infrared image output from the infrared imaging device 10 into an image signal format for output. Many video devices use 8-bit (256 gradations) digital signal formats. The brightness resolution that the human eye can distinguish is around 7-bit (128 gradations). In contrast, in this embodiment, 14-bit (16,384 gradations) is used as the digital signal format for the infrared image output from the infrared imaging device 10.

In this embodiment, the auto gain control unit 25 converts the 14-bit grayscale infrared image output from the infrared imaging device 10 into an 8-bit grayscale infrared image. Specifically, the auto gain control unit 25 maps the 14-bit luminance signal to an 8-bit luminance signal using a linear curve or a histogram flattening curve.

When the contrast of the infrared image output from the infrared imaging device 10 is low, the brightness histogram is concentrated in a specific range. In this case, the auto gain control unit 25 flattens the brightness histogram by lowering the peak of the brightness histogram and widening it on both sides. The auto gain control unit 25 maps the flattened 14-bit brightness signal to an 8-bit brightness signal. This improves the contrast of the infrared image.

Furthermore, if the contrast of the infrared image output from the infrared imaging device 10 is high and a flat area occurs in the brightness histogram, the auto gain control unit 25 maps the 14-bit brightness signal to an 8-bit brightness signal, excluding the flat area. For example, in a scene in which a small, high-temperature object exists against a flat background, the brightness histogram will have a flat shape in the center. In this case, the contrast of the infrared image can be improved by mapping excluding the flat center area.

The infrared image converted to 8-bit grayscale by the auto gain control unit 25 is output to a monitor (not shown) for display. It is also output to a recording medium (not shown) for recording on the recording medium. When recorded on the recording medium, the amount of data may be further compressed according to a predetermined compression encoding standard for recording. It is also output to an image recognition engine (not shown), where a detection process for a predetermined object (e.g., a person) is performed from within the infrared image.

The temperature acquisition unit 26 acquires the environmental temperature of the microbolometer 11 from the infrared imaging device 10. The sensitivity characteristic correction unit 27 can correct the sensitivity characteristic of the microbolometer 11 to the environmental temperature according to the acquired environmental temperature of the microbolometer 11. The sensitivity characteristic is a conversion characteristic for converting an infrared image actually acquired by the microbolometer 11 into a pseudo-infrared image in which the environmental temperature of the microbolometer 11 is considered to be at a constant standard temperature.

The designer derives in advance the relationship between the environmental temperature of the microbolometer 11 and the luminance signal output by the microbolometer 11 based on experiments and simulations. The designer sets the value of the luminance signal when the environmental temperature of the microbolometer 11 is the standard temperature. The designer determines the gain value to be multiplied by the luminance signal at each environmental temperature so that the value of the luminance signal becomes the value at the standard temperature, regardless of the environmental temperature of the microbolometer 11. The generated sensitivity characteristic may be described as a table or as a function.

The sensitivity adjustment unit 24 applies the environmental temperature acquired by the temperature acquisition unit 26 to the sensitivity characteristics described in a table or function to acquire a gain value, and multiplies the acquired gain value by the luminance signals of all pixels of the infrared image to adjust the sensitivity of the infrared image.

As mentioned above, the sensitivity of the microbolometer 11 is highly dependent on the environmental temperature. When adjusting the sensitivity of the luminance signal using one sensitivity curve, the range of the luminance signal after the sensitivity adjustment is greatly expanded. In particular, in vehicle applications, the environmental temperature is wide, and depending on the situation, the environmental temperature may be -40°C or +80°C. Even if the luminance is defined with a resolution of 14 bits (16,384 gradations) as in this embodiment, the luminance signal after the sensitivity adjustment may saturate on the low or high temperature side. In that case, black crush or white blowout occurs in the image.

Therefore, in order to prevent the brightness signal of the infrared image from becoming saturated, one option is to divide one sensitivity curve into multiple curves and move the multiple sensitivity curves closer to the center of the sensitivity range so that the overall sensitivity range is narrowed.

Figure 2 shows an example of a sensitivity curve divided into multiple parts. In Figure 2, three sensitivity curves are prepared: a first sensitivity curve SC1 for high temperatures, a second sensitivity curve SC2 for medium temperatures, and a third sensitivity curve SC3 for low temperatures. When the environmental temperature changes significantly, it becomes necessary to switch the sensitivity curve between the first sensitivity curve SC1 for high temperatures and the second sensitivity curve SC2 for medium temperatures, or between the second sensitivity curve SC2 for medium temperatures and the third sensitivity curve SC3 for low temperatures. When the sensitivity curve is switched, the sensitivity changes significantly.

This change in sensitivity causes the infrared image to change unnaturally, to the extent that it gives the user a sense of discomfort. For example, if the sensitivity suddenly drops due to switching between sensitivity curves, the image will suddenly become dark or the gradation will become coarse. If the auto gain control unit 25 expands the dynamic range of the luminance signal in response to the range reduction of the luminance histogram accompanying the sensitivity reduction, the gradation of the image will become coarse. If the dynamic range is not expanded, the image will become dark. The opposite phenomenon occurs if the sensitivity suddenly increases due to switching between sensitivity curves.

In this embodiment, a mechanism is introduced to suppress degradation of image quality due to the sensitivity difference between the sensitivity curves. In a temperature range in which the sensitivity of the first sensitivity curve SC1 is lower than the sensitivity of the second sensitivity curve SC2 at the temperature of the first switching point between the first sensitivity curve SC1 and the second sensitivity curve SC2, the sensitivity characteristic correction unit 27 corrects the first sensitivity curve SC1 so that the sensitivity of the first sensitivity curve SC1 approaches the sensitivity of the second sensitivity curve SC2 at the temperature of the first switching point. For example, the first sensitivity curve SC1 is corrected so that it matches the sensitivity of the second sensitivity curve SC2 at the temperature of the first switching point.

3 is a diagram showing a plurality of sensitivity curves after the first sensitivity curve SC1 has been corrected. First, in order to eliminate the sensitivity difference between point b on the first sensitivity curve SC1 and point c on the second sensitivity curve SC2 at the first switching temperature, a coefficient α is calculated as shown in the following (Equation 1).
α=c point sensitivity/b point sensitivity (Equation 1)

Next, a coefficient β for defining the relationship between the environmental temperature and the sensitivity in the first sensitivity curve SC1 is calculated.
β=(sensitivity at point a−sensitivity at point b)/ΔT1 (Equation 2)
ΔT1 = temperature at point a - temperature at point b

Since the first sensitivity curve SC1 shown in FIG. 3 can be regarded as a straight line, the coefficient β is derived as the slope of a linear function. Note that if the first sensitivity curve SC1 cannot be regarded as a straight line, the coefficient β is derived as the coefficient of a curve function.

The corrected sensitivity is calculated as shown in the following (Equation 3).
Sensitivity after correction=sensitivity before correction*α−β*ΔT2 (Equation 3)
The sensitivity before correction indicates the sensitivity before correction of the first sensitivity curve SC1.
ΔT2=(measured environmental temperature)−(first switching temperature)
ΔT2>0

In the area of the first sensitivity curve SC1 that is higher than the sensitivity at point d, which is the same as the sensitivity at point c on the second sensitivity curve SC2, the sensitivity is used as is without correction.

The correction of the second sensitivity curve SC2 is similar to the correction of the first sensitivity curve SC1. That is, in a temperature range in which the sensitivity of the second sensitivity curve SC2 is lower than the sensitivity of the third sensitivity curve SC3 at the temperature of the second switching point between the second sensitivity curve SC2 and the third sensitivity curve SC3, the sensitivity characteristic correction unit 27 corrects the second sensitivity curve SC2 so that the sensitivity of the second sensitivity curve SC2 approaches the sensitivity of the third sensitivity curve SC3 at the temperature of the second switching point. For example, the second sensitivity curve SC2 is corrected so that it matches the sensitivity of the third sensitivity curve SC3 at the temperature of the second switching point.

Figure 4 shows multiple sensitivity curves after correction of the first sensitivity curve SC1 and the second sensitivity curve SC2. The solid line portions of the first sensitivity curve SC1, the second sensitivity curve SC2, and the third sensitivity curve SC3 indicate the sensitivities that are actually used.

The sensitivity of the corrected temperature region will be greater than the sensitivity before correction. This increase in sensitivity also increases noise. If this increase in noise makes the noise in the image more noticeable, you can reduce the degree of correction.

5 is a diagram showing a plurality of sensitivity curves after another correction of the first sensitivity curve SC1. When the degree of correction is relaxed, the sensitivity after correction is calculated as shown in the following (Equation 4).
Sensitivity after correction=sensitivity before correction*α−{A*(β*ΔT2)−B} (Equation 4)

The values of coefficient A and intercept B are derived so that the corrected sensitivity of the above (Equation 4) passes through points e and d. In the example shown in Figure 5, intercept B is set to (sensitivity at point c - sensitivity at point b)/2, and coefficient A is set so that the slope becomes steeper as the intercept decreases. The smaller the value of intercept B is set, the smaller the sensitivity difference between the sensitivity curves will be, and the larger the value of intercept B is set, the more the increase in noise can be suppressed.

When the sensitivity characteristics are corrected by the sensitivity characteristic correction unit 27, the sensitivity adjustment unit 24 applies the environmental temperature acquired by the temperature acquisition unit 26 to the corrected sensitivity characteristics to acquire a gain value, and multiplies the acquired gain value by the luminance signals of all pixels of the infrared image to adjust the sensitivity of the infrared image.

The gain value determined based on the sensitivity characteristics corrected by the sensitivity characteristic correction unit 27 and the environmental temperature acquired by the temperature acquisition unit 26 may be used as the gain value of the amplifier 12 in the infrared imaging device 10. In this case, the amplifier 12 is configured as a variable amplifier. The variable amplifier amplifies the analog voltage output from the microbolometer 11 by a gain value for sensitivity adjustment normalized within the input voltage range of the AD converter 13. In this case, the sensitivity adjustment unit 24 uses a fixed gain value.

As described above, according to this embodiment, it is possible to suppress degradation of image quality caused by sensitivity adjustment while enabling sensitivity adjustment for a wide range of environmental temperatures. That is, by using multiple sensitivity curves to narrow the overall sensitivity range, and correcting the sensitivity curves to reduce the sensitivity difference when switching between multiple sensitivity curves, it is possible to suppress degradation of image quality that occurs when the sensitivity curves switch. This makes it possible to suppress a decrease in the user's viewpoint when the sensitivity curve switches. It is also possible to suppress erroneous detection in the image recognition process when the sensitivity curve switches.

In addition, when the sensitivity is adjusted using the amplifier 12 in the infrared imaging device 10, the input voltage range of the AD converter 13 can be narrowed, reducing the circuit area and cost of the AD converter 13.

The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention.

In the above embodiment, an example in which three sensitivity curves are used has been described. In this regard, the number of sensitivity curves is not limited to three, and may be two, or may be four or more.

In the above embodiment, an example was described in which a microbolometer 11 is used as a thermal far-infrared detection element. In this regard, the sensitivity characteristic correction control according to the above embodiment can also be applied to a far-infrared detection element that uses a thermopile or a focusing element.

The sensitivity characteristic correction control according to the above embodiment may be enabled or disabled depending on a predetermined condition. For example, the predetermined condition may be the use of the infrared imaging device 10, the environment of the installation location, the destination, the season, the weather, etc. In the sensitivity characteristic shown in FIG. 2, the second sensitivity curve SC2 for medium temperature allows sensitivity adjustment in the range of 0 to 40°C. If it is predicted that the environmental temperature will not deviate from this range of 0 to 40°C, the sensitivity characteristic correction control is set to disabled.

For example, the sensitivity characteristic correction control is set to enabled for in-vehicle applications, and set to disabled for surveillance camera applications installed indoors in warm regions. Even for in-vehicle applications, if the destination is a warm region and the infrared imaging device 10 is installed in a location where heat is unlikely to build up, the sensitivity characteristic correction control may be set to disabled. In addition, the sensitivity characteristic correction control may be dynamically switched between enabled and disabled depending on seasonal information, time zone information, weather forecast information, etc. obtained from an external server.

1 Infrared imaging system, 10 Infrared imaging device, 11 Microbolometer, 12 Amplifier, 13 AD converter, 14 Thermistor, 15 AD converter, 20 Infrared image processing device, 21 Image acquisition section, 22 Defective pixel correction section, 23 Non-uniformity correction section, 24 Sensitivity adjustment section, 25 Auto gain control section, 26 Temperature acquisition section, 27 Sensitivity characteristic correction section.

Claims (4)

an image acquisition unit that acquires an infrared image from an infrared imaging device;
a temperature acquisition unit that acquires an environmental temperature of an infrared detection element included in the infrared imaging device;
a sensitivity characteristic correction unit that corrects the sensitivity characteristic based on a sensitivity characteristic of the infrared detection element with respect to an environmental temperature and the acquired environmental temperature,
the sensitivity characteristic is defined as a plurality of sensitivity characteristics divided into different temperature ranges so that a luminance signal of the infrared image is not saturated;
When two sensitivity characteristics having adjacent temperature ranges correspond to a higher environmental temperature as a first sensitivity characteristic and a lower environmental temperature as a second sensitivity characteristic,
the sensitivity characteristic correction unit corrects the sensitivity of the first sensitivity characteristic to match the sensitivity of the second sensitivity characteristic at the temperature at the switching point in a temperature range in which the sensitivity of the first sensitivity characteristic is lower than the sensitivity of the second sensitivity characteristic at the temperature at the switching point between the first sensitivity characteristic and the second sensitivity characteristic.
Infrared imaging device. The infrared detection element is a thermal detection element that converts a temperature change caused by absorbing infrared rays into an electrical signal.
The infrared image processing device according to claim 1 . An infrared imaging device;
An infrared image processing device according to claim 1 or 2 ;
An infrared imaging system comprising: acquiring an infrared image from an infrared imaging device;
acquiring an environmental temperature of an infrared detection element included in the infrared imaging device;
and correcting the sensitivity characteristic of the infrared detection element with respect to an environmental temperature based on the acquired environmental temperature,
the sensitivity characteristic is defined as a plurality of sensitivity characteristics divided into different temperature ranges so that a luminance signal of the infrared image is not saturated;
When two sensitivity characteristics having adjacent temperature ranges correspond to a higher environmental temperature as a first sensitivity characteristic and a lower environmental temperature as a second sensitivity characteristic,
the correcting step corrects the sensitivity of the first sensitivity characteristic to match the sensitivity of the second sensitivity characteristic at a temperature at the switching point in a temperature range in which the sensitivity of the first sensitivity characteristic is lower than the sensitivity of the second sensitivity characteristic at a temperature at the switching point between the first sensitivity characteristic and the second sensitivity characteristic.
Infrared imaging methods.

Images