JP7076627B2 - Image processing equipment and thermal image generation systems, as well as programs and recording media - Google Patents

Description

The present invention relates to an image processing apparatus and a thermal image generation system. The present invention also relates to programs and recording media.

A general thermal infrared solid-state image sensor (hereinafter referred to as a thermal image sensor) visualizes infrared rays emitted by a subject, and the infrared rays emitted by the subject are collected by a lens and combined on the image sensor. The difference in temperature rise caused by the image and the absorption of infrared rays by the image sensor is the shade of the image.

A thermal image sensor that can acquire thermal information can acquire information that cannot be acquired by a visible camera. However, in the case of an inexpensive small thermal image sensor, the image resolution, contrast, contour sharpness, and signal-to-noise ratio are small. On the other hand, large thermal image sensors are expensive. It is desired to improve the image quality by using a low-priced thermal image sensor and by image processing.

A method of sharpening a thermal image (an image obtained by photographing with infrared light) has been proposed (see, for example, Patent Document 1).

JP-A-2018-129672 (page 4)

In the case of an inexpensive small thermal image sensor, the method of Patent Document 1 has a problem that noise amplification occurs and visibility is deteriorated when the sharpening process is performed due to insufficient SN ratio.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to make it possible to generate a thermal image that is clear and has a high signal-to-noise ratio.

The image processing apparatus according to one aspect of the present invention is
It has a background image generation unit and an image correction unit.
The background image generation unit is
Identify the intermediate image located in the center when the multi-frame first thermal image acquired by the thermal image sensor in the same field of view or the multi-frame sorted images generated from the first thermal image are arranged in order of brightness. death,
A feature amount that is an index of brightness is calculated for each of the first thermal image and the sorted image.
Of the first thermal image or the sorted image, a plurality of frames of the first thermal image or the sorted image in which the difference in the feature amount from the intermediate image is smaller than a predetermined difference threshold are averaged in the frame direction. To generate an average image,
After sharpening the average image, the skeleton image obtained by extracting the skeleton component is stored in a storage device.
The image correction unit is
The thermal image sensor corrects the second thermal image acquired by imaging in the same field as the first thermal image by using the skeleton image stored in the storage device to correct the thermal image. To generate.

The image processing apparatus of another aspect of the present invention is
It has a background image generation unit and an image correction unit.
The background image generation unit is
Identify the intermediate image located in the center when the multi-frame first thermal image acquired by the thermal image sensor in the same field of view or the multi-frame sorted images generated from the first thermal image are arranged in order of brightness. death,
A feature amount that is an index of brightness is calculated for each of the first thermal image and the sorted image.
Of the first thermal image or the sorted image, a plurality of frames of the first thermal image or the sorted image in which the difference in the feature amount from the intermediate image is smaller than a predetermined difference threshold are averaged in the frame direction. To generate an average image,
The sharpened image obtained by sharpening the average image is stored in a storage device, and the sharpened image is stored in a storage device.
The image correction unit is
The thermal image sensor was obtained by extracting a skeletal component from the sharpened image stored in the storage device with respect to the second thermal image acquired by imaging in the same field as the first thermal image. A corrected thermal image is generated by performing correction using a skeleton image.

According to the present invention, it is possible to generate a thermal image that is clear and has a high signal-to-noise ratio.

It is a figure which shows the schematic structure of the thermal image generation system provided with the image processing apparatus of Embodiment 1 of this invention. It is a functional block diagram of the image processing apparatus of Embodiment 1 of this invention. (A) and (b) are diagrams showing different configuration examples of the sharpening portion of FIG. It is a functional block diagram of the image processing apparatus of Embodiment 2 of this invention. (A) to (d) are diagrams showing different examples of weight tables used in the image processing apparatus of FIG. It is a functional block diagram of the image processing apparatus of Embodiment 3 of this invention. It is a functional block diagram of the image processing apparatus of Embodiment 4 of this invention. It is a figure which shows the process of dividing and coloring an image in the coloring part of the image processing apparatus of FIG. 7. It is a figure which shows the color assignment to the divided image in the coloring part of the image processing apparatus of FIG. It is a functional block diagram of the image processing apparatus of Embodiment 5 of this invention. It is a block diagram which shows the structural example of the computer which executes the processing in the image processing apparatus of Embodiments 1-5.

Embodiment 1.
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows a schematic configuration of a thermal image generation system including the image processing apparatus according to the first embodiment of the present invention.

The infrared generation system shown in FIG. 1 includes a thermal image sensor 1, an image processing device 2, a storage device 3, and a display terminal 4.

The thermal image sensor 1 detects infrared rays radiated from the subject and generates a thermal image showing the temperature distribution of the subject. The infrared ray referred to here is, for example, an electromagnetic wave having a wavelength of 8 μm to 12 μm. The thermal image sensor 1 has a plurality of infrared detection elements arranged one-dimensionally or two-dimensionally. The signal output from each infrared detection element represents the pixel value (pixel value) of the thermal image.

As the infrared detection element, for example, a pyroelectric element can be used. Alternatively, a thermopile-type infrared detection element connected to a thermocouple that causes the Zeebeck effect, a bolometer-type infrared detection element that utilizes a change in resistance value due to a temperature rise, or the like can also be used.

The infrared detection element is not limited to these, and any type of infrared detection element can be used as long as it can detect infrared rays.

FIG. 2 is a functional block diagram of the image processing device 2 of the first embodiment.
The illustrated image processing device 2 has a background image generation unit 21 and an image correction unit 22.

The background image generation unit 21 generates a background image based on a plurality of frames of thermal images output from the thermal image sensor 1.
The multi-frame thermal image used to generate the background image is acquired by the thermal image sensor 1 repeating imaging in the same field of view.

The background image generation unit 21 specifies the order of the size of the pixel values for the pixels at the same position in the thermal image of the above-mentioned plurality of frames, and generates a sorted image of a plurality of frames composed of a set of pixels having the same order. ..
The background image generation unit 21 further identifies a sorted image composed of pixels located at the center when arranged in the order of the size of the pixel values, that is, a set of pixels having an intermediate rank, as an intermediate image.
Since the intermediate image specified in this way is composed of a set of pixels having an intermediate rank, it is located at the center when the sorted images Dc of a plurality of frames are arranged in the order of brightness.

The background image generation unit 21 further calculates a feature amount for each of the sorted images of a plurality of frames, and a plurality of sorted images in which the difference between the feature amounts and the intermediate image is smaller than a predetermined threshold (difference threshold) FDt. Is averaged in the frame direction to generate an average image.
The background image generation unit 21 generates a skeleton image Dg by extracting a skeleton component after sharpening the average image, and stores the generated skeleton image Dg in the storage device 3 as a background image.

Each of the plurality of frames of the thermal image used to generate the background image is referred to as a first thermal image and is represented by the reference numeral Din1.

The image correction unit 22 displays the skeleton image Dg stored in the storage device 3 with respect to the second thermal image Din2 output from the thermal image sensor 1 and obtained by imaging in the same field as the first thermal image Din1. The corrected thermal image Dout is generated by superimposing. The corrected thermal image Dout is sharpened with respect to the second thermal image Din2 and the SN ratio is improved.

The background image generation unit 21 includes a temperature sorting unit 211, a feature amount calculation unit 212, an analysis unit 213, an average image generation unit 214, a sharpening unit 215, and a skeleton component extraction unit 216.

The temperature sorting unit 211 compares the pixels at the same position of the first thermal image Din1 of a plurality of frames, for example, N frames (N is an integer of 2 or more) with each other, and arranges them in the order of the size of the pixel values to specify the order. When arranging in order, the pixel values may be arranged in descending order (descending order) or in ascending order (ascending order).
The temperature sorting unit 211 further generates a multi-frame sorted image Dc composed of a set of pixels having the same rank.
That is, the nth sorted image Dc is composed of a set of pixels having an order of n (n is any one of 1 to N).

The temperature sorting unit 211 further identifies the sorted image Dc composed of the pixels located at the center when arranged in the order of the size of the pixel values, that is, the set of pixels having an intermediate rank, as the intermediate image Dd.

The temperature sorting unit 211 outputs the generated plurality of sorted images Dc together with the information Sdc indicating the respective ranks.
The temperature sorting unit 211 further outputs the information IDd that identifies the intermediate image Dd.

The feature amount calculation unit 212 calculates the feature amount Qf, which is an index of brightness, for each of the sorted images Dc of a plurality of frames. As the feature amount Qf, the average value of the pixel values of the sorted images of each frame, the intermediate value of the pixel values, the maximum value of the pixel values, or the minimum value of the pixel values is calculated.

The analysis unit 213 receives the feature amount Qf of each sorted image from the feature amount calculation unit 212, receives the information IDd for specifying the intermediate image Dd from the temperature sort unit 211, and specifies the high temperature boundary frame Fu and the low temperature boundary frame Fl. ..

The analysis unit 213 determines the image having the largest feature amount among the sorted images having a larger feature amount than the intermediate image Dd and a difference (absolute value) of the feature amount from the intermediate image Dd smaller than the difference threshold value FDt at a high temperature boundary. Specify as frame Fu.
If there is no sorted image that has a larger feature amount than the intermediate image Dd and the difference (absolute value) of the feature amount is equal to or greater than the difference threshold FDt, the image with the largest feature amount among the sorted images is the high temperature boundary frame Fu. Is specified.

The analysis unit 213 further lowers the temperature of the sorted image having the smallest feature amount among the sorted images whose feature amount is smaller than that of the intermediate image Dd and whose difference (absolute value) of the feature amount from the intermediate image Dd is smaller than the difference threshold FDt. It is specified as the boundary frame Fl.
If there is no sorted image whose feature amount is smaller than the intermediate image Dd and the difference (absolute value) of the feature amount is equal to or greater than the difference threshold FDt, the image with the smallest feature amount among the sorted images is the low temperature boundary frame Fl. Is specified.

The analysis unit 213 outputs the information IFu that specifies the high temperature boundary frame Fu and the information IFl that specifies the low temperature boundary frame Fl.

The difference threshold FDt may be stored in the storage device 3 or may be stored in a parameter memory (not shown).

The average image generation unit 214 receives information Sdc indicating the order of each sorted image together with the sorted image Dc from the temperature sorting unit 211, and receives information IFu and a low temperature boundary frame Fl that specify the high temperature boundary frame Fu from the analysis unit 213. Receives the specified information IFl and generates an average image De.

The average image generation unit 214 sets the pixel values of the images of the frames (including the high temperature boundary frame Fu and the low temperature boundary frame Fl) from the high temperature boundary frame Fu to the low temperature boundary frame Fl among the sorted images Dc of a plurality of frames in the frame direction. An average image De is generated on average. "Average in the frame direction" means averaging the pixel values of the pixels at the same position in the images of a plurality of frames.

In generating the average image De, the average image is unsteady by excluding the frame having the feature amount larger than the feature amount of the high temperature boundary frame Fu and the frame having the feature amount smaller than the feature amount of the low temperature boundary frame Fl. It is possible to prevent the influence of a subject appearing on the surface, particularly a heat source (high temperature object) or a low temperature object. The subjects that appear non-stationarily here include people.

The sharpening unit 215 sharpens the average image De to generate a sharpened image Df.
Examples of the sharpening method in the sharpening unit 215 include histogram equalization and the Retinex method.

FIG. 3A shows a configuration example of the sharpening unit 215 that performs sharpening by histogram equalization. The sharpening section 215 shown in FIG. 3A is composed of a histogram equalizing section 2151.
Histogram equalization unit 2151 performs histogram equalization on the average image De. Histogram equalization is a process of calculating the distribution of pixel values in the entire image and converting the pixel values so that the distribution of pixel values becomes the distribution of a desired shape.
Histogram equalization may be contrast limited adaptive Histogram Equalization.

FIG. 3 (b) shows a configuration example of the sharpening unit 215 that sharpens by the Retinex method.
The sharpening section 215 shown in FIG. 3B has a filter separating section 2152, adjusting sections 2153 and 2154, and a combining section 2155.

The filter separation unit 2152 separates the input average image De into a low frequency component Del and a high frequency component Deh.
The adjusting unit 2153 adjusts the magnitude of the pixel value by multiplying the low frequency component Del by the first gain.
The adjusting unit 2154 adjusts the size of the pixel value by multiplying the high frequency component Deh by the second gain. The second gain is larger than the first gain.
The synthesizing unit 2155 synthesizes the outputs of the adjusting units 2153 and 2154. The image obtained as a result of the composition has an enhanced high frequency component.

The skeleton component extraction unit 216 extracts the skeleton component from the sharpening image Df output from the sharpening unit 215, and generates a skeleton image Dg composed of the extracted skeleton components.
The skeleton component is a component representing the global structure of an image, and includes an edge component and a flat component (a slowly changing component) in the image. For the extraction of the skeletal component, for example, the total variational norm minimization method can be used.

The background image generation unit 21 transmits the skeleton image Dg as a background image to the storage device 3 and stores it.

The image correction unit 22 corrects the second thermal image Din2 output from the thermal image sensor 1 using the skeleton image Dg stored in the storage device 3, and generates and outputs a corrected thermal image Dout. ..
As described above, the second thermal image Din2 is obtained by imaging in the same field of view as the first thermal image Din1. The second thermal image Din2 may be obtained by imaging at a different imaging time from the first thermal image Din1, or an image of one frame of the first thermal image Din1 may be used as the second thermal image. good.

In the example of FIG. 2, the image correction unit 22 has a superimposing unit 221.
The superimposing unit 221 generates a corrected thermal image Dout by superimposing the skeleton image Dg on the second thermal image Din2. Superimposition is performed, for example, by weighted addition.
When adding the skeleton image Dg, the gain may be multiplied by the skeleton image Dg so that the components of the skeleton image Dg become clearer.

Such weighted addition is expressed by the following equation (1).
P _Dout = P _Din2 + P _Dg × g formula (1)
In equation (1)
P _Din2 is the pixel value of the second thermal image Din2,
P _Dg is the pixel value of the skeleton image Dg,
g is the gain for the skeleton image Dg,
P _Dout is a pixel value of the corrected thermal image Dout.

The above is a description of the operation of the image processing apparatus according to the first embodiment.
In the image processing apparatus of the first embodiment, a skeletal image with less noise and high contrast is generated from the first thermal image of a plurality of frames and stored as a background image, and the stored skeletal image is combined with the second thermal image. Therefore, it is possible to generate a thermal image having a high SN ratio and a high time resolution.

Further, instead of using the average image as it is as a background image, only the skeleton component of the average image is extracted and stored as a background image, and the temperature information of the second thermal image is saved by adding it to the second thermal image. At the same time, information on the background structure can be added.

Further, in generating the background image, by excluding the thermal image of the frame in which the difference in the feature amount from the intermediate image Dd is equal to or larger than the difference threshold FDt, the subject appearing unsteadily in the background image, particularly the heat source (high temperature). It can be prevented from being affected by an object) or a cold object.

Embodiment 2.
FIG. 4 is a functional block diagram of the image processing device 2a according to the second embodiment of the present invention.
The image processing device 2a shown in FIG. 4 is generally the same as the image processing device 2 of FIG. 2, but instead of the background image generation unit 21 and the image correction unit 22, the background image generation unit 21b and the image correction unit 22b are used. Be prepared.
The background image generation unit 21b is generally the same as the background image generation unit 21, but includes a threshold value generation unit 217.
The image correction unit 22b is generally the same as the image correction unit 22, but includes a weight determination unit 222.

The threshold value generation unit 217 generates a weight determination threshold value Th, transmits the threshold value Th to the storage device 3, and stores the threshold value Th. The threshold generation unit 217 obtains, for example, the average value or the intermediate value of the pixel values of the average image De output from the average image generation unit 214, and determines the threshold value Th based on the calculated average value or the intermediate value. The average value or the intermediate value of the pixel values of the average image De means the average value or the intermediate value of the pixel values of the pixels located in the whole or the main part of the average image De.
The relationship between the threshold value Th and the above average value or the intermediate value is determined based on experience or experiment (simulation).

The threshold value Th may be set to a value higher than the above average value or the intermediate value. In this case, the value obtained by adding the difference threshold value FDt to the above average value or the intermediate value may be defined as the threshold value Th. In such a case, the difference threshold FDt is also notified to the threshold generation unit 217.
The threshold value generation unit 217 transmits the generated threshold value Th to the storage device 3 and stores it.

The weight determination unit 222 creates a weight table based on the threshold value Th stored in the storage device 3, refers to the created weight table, and generates a composite weight w based on the pixel value of the second thermal image Din2. do.
Examples of the weight table created by the weight determination unit 222 based on the threshold value Th are shown in FIGS. 5 (a) and 5 (b).

In the examples shown in FIGS. 5A and 5B, in the range where the pixel value P _Din2 of the second thermal image Din2 is from 0 to the threshold value Th, the combined weight w is kept at 1 and the pixel value P _Din2 is the threshold value. In a range larger than Th, the combined weight w gradually decreases as the pixel value P _{Din 2} increases.

By using the weighting table shown in FIGS. 5A or 5B, the weighted addition rate of the skeleton image Dg can be reduced only when the pixel value of the second thermal image Din2 is higher than the threshold value Th. can.

In the above example, the weight determination unit 222 creates the weight table using the threshold Th, but the weight table may be created without using the threshold Th. Examples of the weight table created without using the threshold Th are shown in FIGS. 5 (c) and 5 (d).
In the examples shown in FIGS. 5 (c) and 5 (d), when the pixel value P _Din2 of the second thermal image Din2 is 0, the combined weight w is 1, and as the pixel value P _Din2 increases, the combined weight w Gradually becomes smaller. Even in such a weight table, the addition ratio of the skeleton image Dg can be reduced in the range where the pixel value P _{Din 2} is large.

In short, the weight table may be such that the composite weight w becomes smaller as the pixel value of the second thermal image Din2 becomes larger.

When creating the weight table shown in FIGS. 5 (c) or 5 (d), the background image generation unit 21a does not need to include the threshold value generation unit 217 (hence, it is the same as the background image generation unit 21 of FIG. 2). The weight determination unit 222 does not need to read the threshold value Th from the storage device 3.

Embodiment 3.
FIG. 6 is a functional block diagram of the image processing device 2c according to the third embodiment of the present invention.
The image processing device 2c shown in FIG. 6 is generally the same as the image processing device 2 of FIG. 2, but instead of the background image generation unit 21 and the image correction unit 22, the background image generation unit 21c and the image correction unit 22c are used. Be prepared.
The background image generation unit 21c is generally the same as the background image generation unit 21 of FIG. 2, but does not include the skeleton component extraction unit 216 of FIG. 2, and the sharpening image Df output from the sharpening unit 215 is used as a background image. Is stored in the storage device 3.

The image correction unit 22c reads out the sharpened image Df stored in the storage device 3, generates a skeleton image Dg by extracting the skeleton component, and corrects the second thermal image Din2 using the skeleton image Dg. ..
That is, in the image processing apparatus 2c shown in FIG. 6, the extraction of the skeleton component is performed not by the background image generation unit but by the image correction unit.

Specifically, the image correction unit 22c has a skeleton component extraction unit 223 and a superimposing unit 221.
The skeleton component extraction unit 223 reads the sharpened image Df stored in the storage device 3 and extracts the skeleton component to generate the skeleton image Dg.
The superimposing unit 221 corrects the second thermal image Din2 by superimposing the skeleton image Dg on the second thermal image Din2, and generates the corrected thermal image Dout.

In the case of the configuration shown in FIG. 6, by reading the sharpened image Df from the storage device 3 and displaying it, it is possible to easily determine whether or not the sharpened image Df contains a heat source.

Embodiment 4.
FIG. 7 is a functional block diagram of the image processing device 2d according to the third embodiment of the present invention.
The image processing device 2d shown in FIG. 7 is generally the same as the image processing device 2b of FIG. 4, but instead of the background image generation unit 21b and the image correction unit 22b, the background image generation unit 21d and the image correction unit 22d are used. Be prepared.
The background image generation unit 21d is generally the same as the background image generation unit 21b, but includes a threshold value generation unit 217d instead of the threshold value generation unit 217.

The threshold generation unit 217d obtains the average value or the intermediate value of the pixel values of the average image De output from the average image generation unit 214, and based on the calculated average value or the intermediate value, in addition to the weight determination threshold Th, the image. The high temperature threshold Tu and the low temperature threshold Tl for division are generated, and the generated thresholds Th, Tu and Tl are transmitted to the storage device 3 and stored.

The high temperature threshold value Tu and the low temperature threshold value Tl are used for image division.
The high temperature threshold value Tu is obtained by adding the difference threshold value FDt to the average value or the intermediate value of the pixel values of the average image De.
The low temperature threshold value Tl is obtained by subtracting the difference threshold value FDt from the average value or the intermediate value of the pixel values of the average image De.

When the high temperature threshold value Tu is generated as described above, the weight determination threshold value Th may be the same as the high temperature threshold value Tu.

The image correction unit 22d divides the second thermal image Din2 into a high temperature region, an intermediate temperature region, and a low temperature region using the high temperature threshold Tu and the low temperature threshold Tl read from the storage device 3, and colors each region. A color image Dh is generated by performing and synthesizing, and a corrected thermal image Dout is generated and output by synthesizing the color image Dh and the skeleton image Dg taken out from the storage device 3.
The corrected thermal image Dout in this case is a color image colored according to the temperature of each part.
The image correction unit 22d has a weight determination unit 222, a coloring unit 224, and a superimposing unit 221d.

The weight determination unit 222 creates a weight table and determines the weight as described with respect to the configuration of FIG.
When creating the weight table shown in FIG. 5A or FIG. 5B, it is necessary to use the threshold value Th. As described above, the threshold Th may be the same as the high temperature threshold Tu. In that case, the high temperature threshold value Tu stored in the storage device 3 can be read out and used as the threshold value Th for creating the weight table.

When creating the weight table shown in FIGS. 5A or 5B, the weighted addition ratio of the skeleton image Dg can be reduced only when the pixel value of the second thermal image Din2 is higher than the threshold value Th. can. When the threshold value Th is the same as the high temperature threshold value Tu, the rate of addition of the skeleton image Dg can be reduced only when the pixel value P _Din2 of the second thermal image Din2 belongs to the high temperature region.

As described with respect to the configuration of FIG. 4, the weight table may be that shown in FIGS. 5 (c) or 5 (d).
In short, the weight table may be such that the composite weight w becomes smaller as the pixel value of the second thermal image Din2 becomes larger.

As shown in FIGS. 8 and 9, the coloring unit 224 divides the second thermal image Din2 into a high temperature region, an intermediate temperature region, and a low temperature region using the high temperature threshold Tu and the low temperature threshold Tl, and for each region. Coloring is performed, and a color image Dh is generated by synthesizing the colored images. The color image Dh is represented by, for example, a red (R), green (G), and blue (B) signal.

That is, each pixel of the second thermal image Din2 is determined to belong to the high temperature region if its pixel value is larger than the high temperature threshold value Tu, and if the pixel value is equal to or less than the high temperature threshold value Tu and equal to or higher than the low temperature threshold value Tl. , It is determined that it belongs to the intermediate temperature region, and if the pixel value is smaller than the low temperature threshold value Tl, it is determined that it belongs to the low temperature region.
In the example shown in FIG. 8, the pixels constituting the light emitting portion 101 of the street light, the automobile 103, and the person 105 are determined to belong to the high temperature region, and the pixels constituting the road marking 107 of the road belong to the intermediate temperature region. It is determined that the pixels constituting the pillar 109 of the street light belong to the low temperature region.

The coloring unit 224 assigns colors in different ranges, that is, first, second, and third ranges to the high temperature region, the intermediate temperature region, and the low temperature region, and in each region, the colors in the range assigned to the region are assigned. Of these, a color corresponding to the pixel value is assigned to each pixel.
At that time, in the boundary portion between the high temperature region, the intermediate temperature region, and the low temperature region, the above color assignment and the pixel value to the high temperature region, the intermediate temperature region, and the low temperature region are applied so that the color change is continuous. It is desirable to assign different colors.

For example, as shown in FIG. 9, the hue range centered on red in the high temperature region (for example, from the center of magenta (center in the hue direction) to the center of yellow), and the hue centered on green in the intermediate temperature region. A range (from the center of yellow to the center of cyan) and a hue range centered on blue (from the center of cyan to the center of magenta) are assigned to the low temperature region, and in each region, the hue assigned to each pixel value is assigned. Assign a range of colors.

The superimposing unit 221d weights and adds the color image Dh and the skeleton image Dg read from the storage device 3 by using the combined weight w.
The color image Dh is represented by R, G, B signals, that is, signals of 3 channels, whereas the skeleton image Dg is represented by signals of 1 gray channel.
The skeleton image Dg is added to the luminance component Dhy of the color image Dh.

In one example of processing, a color image Dh is converted into a brightness component Dhy and a color component such as a color difference component Dhcb or Dhcr, a skeleton image Dg is added to the brightness component Dhy, and the added brightness component Djy and the above color component are added. For example, the values of the R, G, and B components of the corrected thermal image are obtained by performing inverse conversion from the color difference components Dhcb and Dhcr to R, G, and B.

The addition of the above skeleton image Dg is expressed by the following formula.
P _Djy = P _Dhy + P _Dg * g * w equation (2)
In equation (2)
P _Dhy is the value of the luminance component Dhy of the color image Dh,
P _Dg is the pixel value of the skeleton image Dg,
g is the gain for the skeleton image Dg,
w is the synthetic weight,
P _Djy is a value of the luminance component Djy obtained as a result of the addition.

In another example of processing, when the color image Dh is composed of signals of three channels of R, G, and B, the skeleton image Dg is added to each channel.
The addition in this case is expressed by the following equations (3a) to (3c).
P _Rout = P _Rin + P _Dg * g * w equation (3a)
P _Gout = P _Gin + P _Dg * g * w equation (3b)
P _Bout = P _Bin + P _Dg * g * w equation (3b)

In equations (3a) to (3c),
P _Rin is the value of the signal Rin (value of the red component) of the R channel of the color image Dh,
P _Gin is the value of the signal Gin (value of the green component) of the G channel of the color image Dh,
P _Bin is the value of the signal Bin of the B channel of the color image Dh (the value of the blue component),
P _Dg is the pixel value of the skeleton image Dg,
g is the gain for the skeleton image Dg,
w is the synthetic weight,
P _Rout is the value of the signal Rout (red component value) of the R channel obtained as a result of addition.
P _Gout is the value of the signal Gout of the G channel (value of the green component) obtained as a result of the addition.
P _Bout is a value (blue component value) of the signal Bout of the B channel obtained as a result of the addition.

The above is a description of the operation of the image processing apparatus according to the fourth embodiment.
The same effect as that of the first embodiment can be obtained in the fourth embodiment.

In the fourth embodiment, the brightness component of the color image generated by coloring the second heat image and the skeleton image Dg are combined, so that the information indicating the heat source and the skeleton image are visually separated. It is possible to improve the visibility of the heat source. That is, when the second heat image Din2 and the skeleton image Dg are combined without coloring, the information indicating the heat source contained in the second heat image Din2 may be buried in the skeleton image Dg, but the second heat Coloring the image can prevent such a situation from occurring.

Further, by using the weight table shown in FIGS. 5A or 5B, when the pixel value of the second thermal image Din2 is equal to or less than the threshold Th when the image is corrected, the composite weight is increased to increase the skeleton. The skeleton image Dg becomes the second thermal image Din2 by sufficiently correcting the second thermal image Din2 by the image Dg and reducing the composite weight when the pixel value of the second thermal image Din2 is higher than the threshold Th. It has the effect of suppressing the addition of heat to the region where the heat source exists in a large proportion and improving the visibility.

Further, regardless of the overall brightness of the second thermal image Din2 (for example, the average value of the pixel values), the color assignment to the pixel value is not fixed, but the second thermal image Din2 is in the high temperature region and the intermediate temperature region. By dividing into low temperature areas and assigning different colors to each area for coloring, relatively high temperature areas in the image are always colored with high temperature colors (colors assigned to high temperatures). In the thermal image, the relatively low temperature part can always be expressed by the low temperature color (color assigned to the low temperature). For example, if the thermal image contains a temperature offset, a fixed color assignment to the pixel values could give the cold region a color that represents intermediate temperature, but this is prevented. effective.

That is, when displaying a thermal image in color, for example, it is one method of coloring processing that a high temperature subject is displayed in red, a low temperature subject is displayed in blue, and an intermediate temperature subject is displayed in green. If offsets are included, for example, low to mid-temperature can be displayed in green.
By dividing the thermal image into a high temperature region, an intermediate temperature region, and a low temperature region and coloring each region, it is possible to prevent the occurrence of such a situation.

In the fourth embodiment, as in the first embodiment, the background image generation unit 21d performs sharpening and extraction of the skeleton component, the skeleton image is stored in the storage device 3, and the image correction unit 22d is stored in the storage device 3. The stored skeleton image is read out and used for correction of the second thermal image.

Also in the fourth embodiment, as described in the third embodiment, the background image generation unit 21d stores the sharpening image Df obtained by the sharpening in the storage device 3, and the image correction unit 22d stores the sharpening image Df. A skeletal image may be generated by reading out the sharpened image Df stored in the apparatus 3 and extracting the skeletal component, and the generated skeletal image may be used for correction of the second thermal image.

Further, in the fourth embodiment, the weight determination unit 222 may be omitted and weighted addition may be performed using a composite weight of a constant value.

Embodiment 5.
FIG. 10 is a functional block diagram of the image processing device 2e according to the fifth embodiment.
The image processing device 2e shown in FIG. 10 is generally the same as the image processing device 2 of FIG. 2, but includes a background image generation unit 21e instead of the background image generation unit 21.

The background image generation unit 21e is generally the same as the background image generation unit 21, but instead of the temperature sorting unit 211, the feature amount calculation unit 212, the analysis unit 213, and the average image generation unit 214, the temperature sorting unit 211e, the feature It includes a quantity calculation unit 212e, an analysis unit 213e, and an average image generation unit 214e.

The feature amount calculation unit 212e calculates the feature amount Qe, which is an index of brightness, for each of the first thermal images Din1 of a plurality of frames, that is, for the first thermal image of each frame.
As the feature amount Qe, the average value of the pixel values of each frame, the intermediate value of the pixel values, the maximum value of the pixel values, or the minimum value of the pixel values is calculated.

The temperature sorting unit 211e receives the feature amount Qe calculated by the feature amount calculation unit 212e, and specifies the order of the magnitude of the feature amount Qe for the first thermal image Din1 of a plurality of frames. When arranging in order, the features may be arranged in descending order (descending order) or in ascending order (ascending order).

The temperature sorting unit 211e further identifies the first thermal image Din1 which is located in the center when arranged in the order of the size of the feature quantity Qe, that is, has an intermediate rank, as the intermediate image Dd.

The temperature sorting unit 211e outputs information Sdin indicating the order of each of the first thermal images Din1 of a plurality of frames.
The temperature sorting unit 211e also outputs the information IDd that identifies the intermediate image Dd.

The analysis unit 213e receives the feature amount Qe of each first thermal image from the feature amount calculation unit 212e, receives the information IDd for specifying the intermediate image Dd from the temperature sort unit 211e, and receives the high temperature boundary frame Fu and the low temperature boundary frame Fl. Identify.

The analysis unit 213e selects the image having the largest feature amount among the first thermal images having a feature amount larger than that of the intermediate image Dd and a difference (absolute value) of the feature amount from the intermediate image Dd smaller than the difference threshold value FDt. It is specified as a high temperature boundary frame Fu.
If there is no first thermal image in which the feature amount is larger than the intermediate image Dd and the difference (absolute value) of the feature amount is equal to or greater than the difference threshold FDt, the image having the largest feature amount among the first thermal images is used. It is specified as a high temperature boundary frame Fu.

The analysis unit 213e further has a feature amount smaller than that of the intermediate image Dd, and the difference (absolute value) of the feature amount from the intermediate image Dd is smaller than the difference threshold FDt. Is specified as the low temperature boundary frame Fl.
When there is no first thermal image in which the feature amount is smaller than the intermediate image Dd and the difference (absolute value) of the feature amount is equal to or larger than the difference threshold FDt, the image having the smallest feature amount among the first thermal images is used. It is specified as the low temperature boundary frame Fl.

The analysis unit 213e outputs the information IFu that specifies the high temperature boundary frame Fu and the information IFl that specifies the low temperature boundary frame Fl.

As described in the first embodiment, the difference threshold value FDt may be stored in the storage device 3 or may be stored in a parameter memory (not shown).

The average image generation unit 214e receives the input first thermal image Din1, receives information Sdin indicating the order of the first thermal image of each frame from the temperature sorting unit 211e, and receives the high temperature boundary frame Fu from the analysis unit 213e. The information IFl that identifies the specified information IFu and the low temperature boundary frame Fl are received, and the average image De is generated.

The average image generation unit 214e frames the pixel values of the images of the frames (including the high temperature boundary frame Fu and the low temperature boundary frame Fl) from the high temperature boundary frame Fu to the low temperature boundary frame Fl among the first thermal image Din1 of a plurality of frames. An average image De is generated by averaging in the direction.

In generating the average image De, the average image is unsteady by excluding the frame having the feature amount larger than the feature amount of the high temperature boundary frame Fu and the frame having the feature amount smaller than the feature amount of the low temperature boundary frame Fl. It is possible to prevent the influence of a subject appearing on the surface, particularly a heat source (high temperature object) or a low temperature object. The subjects that appear non-stationarily here include people.

The processing in the sharpening unit 215 and the skeletal component extraction unit 216 is the same as that described in the first embodiment.

As described above, the background image generation unit 21e calculates the feature amount Qe for each of the first thermal images Din1 of the plurality of frames, and when the thermal images of the plurality of frames are arranged in the order of the size of the feature amount Qe, the center. A plurality of thermal images whose difference in feature amount from the thermal image located in is smaller than a predetermined threshold (difference threshold) FDt are averaged in the frame direction to generate an average image De, and the average image is sharpened. After that, a skeletal image Dg is generated by extracting the skeletal component, and the generated skeletal image Dg is stored in the storage device 3 as a background image.

The image correction unit 22 is the same as the image correction unit 22 of the first embodiment, and operates in the same manner.

In the fifth embodiment, the temperature is sorted according to the feature amount for each frame, so that the process is relatively simple.

In addition to the above-mentioned modifications, various modifications can be further added to the image processing apparatus of each embodiment described above.
It is also possible to combine the features of each embodiment with the features of other embodiments.
For example, although the second embodiment has been described as a modification to the first embodiment, the same modification can be applied to the third embodiment.
Further, although the fifth embodiment has been described as a modification to the first embodiment, the same modification can be applied to the second to fourth embodiments.

The image processing devices 2, 2b, 2c, 2d or 2e described in the first to fifth embodiments may be partially or wholly composed of a processing circuit.
For example, the functions of each part of the image processing apparatus may be realized by separate processing circuits, or the functions of a plurality of parts may be collectively realized by one processing circuit.
The processing circuit may be composed of dedicated hardware or software, that is, a programmed computer.
Of the functions of each part of the image processing device, a part may be realized by dedicated hardware and a part may be realized by software.

FIG. 11 shows an example of a configuration in which a computer 200 including a single processor realizes all the functions of the image processing devices 2, 2b, 2c, 2d, or 2e according to the above embodiment. 1. Shown together with the storage device 3 and the display terminal 4.

In the illustrated example, the computer 300 has a processor 310 and a memory 320.
The memory 320 or the storage device 3 stores a program for realizing the functions of each part of the image processing device.

The processor 310 uses, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, a DSP (Digital Signal Processor), or the like.

The memory 320 is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ReadOnly Memory), an EEPROM (Electrically Memory Disk, etc.) Alternatively, a photomagnetic disk or the like is used.

The processor 310 realizes the function of the image processing device by executing the program stored in the memory 320 or the storage device 3. If the program is stored in the storage device 3, it may be loaded into the memory 320 and then executed.
The functions of the image processing device include control of display on the display terminal 4, writing of information on the storage device 3, and reading of information from the storage device 3 as described above.

The above processing circuit may be attached to the thermal image sensor 1. That is, the image processing devices 2, 2b, 2c, 2d or 2e can also be mounted on a processing circuit attached to the thermal image sensor. Alternatively, the image processing devices 2, 2b, 2c, 2d or 2e can be mounted on a cloud server that can be connected to the thermal image sensor 1 via a communication network.
Further, the storage device 3 may be a storage area on a server on the cloud.

At least one of the image processing device and the storage device may be mounted in a communication mobile terminal, for example, a smartphone or a remote controller.
The thermal image generation system including the image processing device may be applied to home appliances, in which case at least one of the image processing device and the storage device is contained in a HEMS (Home Energy Management System) controller. It may be implemented.

The display terminal may also be mounted on a communication terminal, for example, a smartphone or a HEMS (Home Energy Management System) controller.

The thermal image generation system including the image processing apparatus and the image processing apparatus of the present invention has been described above. The image processing method performed by the above image processing apparatus also forms a part of the present invention. A computer-readable recording medium on which a program and a computer-recorded program for causing a computer to perform processing in the above-mentioned image processing apparatus or image processing method also form a part of the present invention.

Although embodiments of the present invention have been described, the present invention is not limited to these embodiments.

1 Thermal image sensor, 2,2b, 2c, 2d, 2e image processing device, 3 Storage device, 4 Display terminal, 21,21b, 21c, 21d, 21e Background image generation unit, 22,22b, 22c, 22d Image correction unit , 211,211e temperature sorting unit, 212,212e feature calculation unit, 213,213e analysis unit, 214,214e average image generation unit, 215 sharpening unit, 216 skeleton component extraction unit, 217,217d threshold generation unit, 221,221d Superimposition part, 222 weight determination part, 223 skeleton component extraction part, 224 coloring part.

Claims (16)

It has a background image generation unit and an image correction unit.
The background image generation unit is
Identify the intermediate image located in the center when the multi-frame first thermal image acquired by the thermal image sensor in the same field of view or the multi-frame sorted images generated from the first thermal image are arranged in order of brightness. death,
A feature amount that is an index of brightness is calculated for each of the first thermal image and the sorted image.
Of the first thermal image or the sorted image, a plurality of frames of the first thermal image or the sorted image in which the difference in the feature amount from the intermediate image is smaller than a predetermined difference threshold are averaged in the frame direction. To generate an average image,
After sharpening the average image, the skeleton image obtained by extracting the skeleton component is stored in a storage device.
The image correction unit is
The thermal image sensor corrects the second thermal image acquired by imaging in the same field as the first thermal image by using the skeleton image stored in the storage device to correct the thermal image. An image processing device that produces. It has a background image generation unit and an image correction unit.
The background image generation unit is
Identify the intermediate image located in the center when the multi-frame first thermal image acquired by the thermal image sensor in the same field of view or the multi-frame sorted images generated from the first thermal image are arranged in order of brightness. death,
A feature amount that is an index of brightness is calculated for each of the first thermal image and the sorted image.
Of the first thermal image or the sorted image, a plurality of frames of the first thermal image or the sorted image in which the difference in the feature amount from the intermediate image is smaller than a predetermined difference threshold are averaged in the frame direction. To generate an average image,
The sharpened image obtained by sharpening the average image is stored in a storage device, and the sharpened image is stored in a storage device.
The image correction unit is
The thermal image sensor was obtained by extracting a skeletal component from the sharpened image stored in the storage device with respect to the second thermal image acquired by imaging in the same field as the first thermal image. An image processing device that generates a corrected thermal image by performing correction using a skeletal image. The background image generation unit is
The order of the size of the pixel values is specified for the pixels at the same position in the first thermal image of the plurality of frames.
A plurality of frames of images each composed of a set of pixels having the same rank are generated as the sorted image.
Among the sorted images, the sorted image composed of a set of pixels having an intermediate rank is specified as the intermediate image.
The image processing apparatus according to claim 1 or 2, wherein the feature amount is calculated for each of the sorted images of the plurality of frames. The background image generation unit is
The feature amount was calculated for each of the first thermal images of the plurality of frames.
The first or second aspect of claim 1 or 2, wherein when the first thermal image of the plurality of frames is arranged in the order of the size of the feature amount, the first thermal image located at the center is specified as the intermediate image. Image processing equipment. The image correction unit is
The image processing apparatus according to any one of claims 1 to 4, wherein the corrected thermal image is generated by weighting and adding the second thermal image and the skeleton image. The image correction unit is
When correcting the second thermal image using the skeleton image,
When the pixel value of the second thermal image is equal to or greater than the weight determination threshold, the larger the pixel value of the second thermal image is, the smaller the weighting of the skeleton image is and the weighting is performed. The image processing apparatus according to claim 5. The image correction unit generates a color image by coloring the second thermal image, and generates the corrected thermal image by correcting the color image using the skeleton image. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is characterized. The image correction unit is
The color image is converted into a luminance component image and a color component image, and then
A corrected luminance component image is generated by weighting and adding the luminance component image and the skeleton image.
The image processing apparatus according to claim 7, wherein the corrected thermal image is generated by converting the corrected luminance component image and the color component image into a color image. The image correction unit is
The second thermal image is divided into a high temperature region, an intermediate temperature region, and a low temperature region using a high temperature threshold value and a low temperature threshold value for image segmentation, and each region is colored and synthesized to obtain the color image. The image processing apparatus according to claim 7 or 8, wherein the image processing apparatus is generated. The image correction unit is
Assign different ranges of colors to the high temperature region, the intermediate temperature region, and the low temperature region.
In each area, out of the colors in the range assigned to the area, the color corresponding to the pixel value is assigned to each pixel.
In the high temperature region, the intermediate temperature region, and the boundary portion between the low temperature regions, the allocation and the pixel value to the high temperature region, the intermediate temperature region, and the low temperature region are set so that the color change is continuous. The image processing apparatus according to claim 9, wherein the image processing apparatus is assigned according to the requirements. The background image generation unit is
A value obtained by adding the difference threshold value to the average value or the median value of the pixel values of the average image is used as the high temperature threshold value.
The image processing apparatus according to claim 9 or 10, wherein a value obtained by subtracting the difference threshold value from the average value or an intermediate value of the pixel values of the average image is used as the low temperature threshold value. The image processing apparatus according to any one of claims 1 to 11, wherein the image processing apparatus is mounted on a processing circuit attached to the thermal image sensor. The image processing apparatus according to any one of claims 1 to 11, wherein the image processing apparatus is mounted on a cloud server that can be connected to the thermal image sensor via a communication network. A thermal image generation system including the image processing device according to any one of claims 1 to 13, the thermal image sensor, and the storage device. A program for causing a computer to perform processing in the image processing apparatus according to any one of claims 1 to 13. A computer-readable recording medium on which the program according to claim 15 is recorded.

Images