JP7515622B2 - Thermal image processing device, thermal image processing system, and thermal image processing method - Google Patents

Description

The present disclosure relates to a thermal image processing device, a thermal image processing system, and a thermal image processing method that provide images that protect privacy in electronic devices equipped with thermal sensors.

Conventionally, air conditioners are known that acquire thermal images and use the acquired thermal images to control air conditioning. Such thermal images may contain areas where privacy protection is desirable. Patent Document 1 discloses a thermal image processing device that determines privacy areas from thermal images acquired by a thermal image sensor and corrects the areas.

The technology described in Patent Document 1 defines privacy areas in thermal images as human faces, detects privacy areas using image processing technology such as face detection, and conceals the detected areas. The technology described in Patent Document 1 can protect privacy information that could lead to the identification of individuals by concealing human faces and people in thermal images.

JP 2020-79784 A

However, while the above-mentioned conventional technology can protect the privacy of individuals, it does not protect privacy information related to living spaces, such as the state of a room. For example, there is a problem in that information about the arrangement of objects in living spaces where people spend a relatively long time, such as sofas or beds, and information about people's movements in those areas, is not protected.

This disclosure has been made in consideration of the above, and aims to obtain a thermal image processing device that can protect privacy information related to living space.

In order to solve the above-mentioned problems and achieve the object, the thermal image processing device according to the present disclosure includes a privacy region calculation unit that counts a threshold exceedance count indicating the number of times that the pixel value exceeds a temperature threshold within a certain time for each pixel of multiple thermal images captured at the same location at different times, and determines whether or not the pixel is a pixel that requires correction using the threshold exceedance count, and a corrected thermal image generation unit that corrects the thermal image using the result of the judgment by the privacy region calculation unit , and the result of the judgment is a flag indicating, for each pixel, with a binary value of 0 or 1 whether or not the pixel is a pixel that requires correction, the corrected thermal image generation unit includes a correction region information storage unit that stores the result of the judgment for all pixels of the thermal image as correction region information, the correction region information storage unit stores the result of the judgment in the correction region information storage unit as correction region information, the correction region information storage unit stores the correction region information before the update until the correction region information is updated by the privacy region calculation unit, and the corrected thermal image generation unit corrects the thermal images acquired sequentially using the correction region information stored in the correction region information storage unit .

This disclosure has the effect of protecting privacy information related to living spaces.

FIG. 1 is a diagram showing a configuration example of a thermal image processing system according to a first embodiment; FIG. 1 is a diagram showing an example of an air conditioner equipped with a thermal image processing device according to a first embodiment; 1 is a flowchart showing an example of a data processing method of a thermal image acquisition unit according to a first embodiment. 1 is a flowchart showing an example of a data processing method of a thermal image storage unit according to a first embodiment. FIG. 1 is a conceptual diagram of thermal images stored in a time series in a thermal image storage unit according to a first embodiment; A flowchart showing an example of a data processing method of a privacy region calculation unit according to the first embodiment. Graph showing an example of temperature change when the temperature value of an arbitrary pixel (x, y) in a thermal image is plotted over time. A flowchart showing an example of a method for calculating correction area information by a privacy area calculation unit according to the first embodiment. 1 is a flowchart showing an example of a data processing method of a corrected thermal image generating unit according to a first embodiment. FIG. 1 is a conceptual diagram showing an example of correction area information calculated by a privacy area calculation unit according to a first embodiment; FIG. 1 is a conceptual diagram showing an example of a thermal image correction processing method of a thermal image correction unit according to a first embodiment; FIG. 1 is a diagram showing a configuration example of a processing circuit according to an embodiment of the present invention; FIG. 13 is a diagram showing a configuration example of a thermal image processing system in which a thermal image storage unit, a privacy region calculation unit, and a modified thermal image generation unit according to a second embodiment are implemented in a cloud. FIG. 13 is a diagram showing a configuration example of a thermal image processing system in which a thermal image storage unit and a privacy region calculation unit according to a second embodiment are implemented in a cloud and a corrected thermal image generation unit is implemented in an image display device. FIG. 13 is a diagram showing a configuration example of a thermal image processing system in which a thermal image storage unit and a privacy region calculation unit according to a second embodiment are implemented in a cloud and a corrected thermal image generation unit is implemented in an air conditioner.

Below, the thermal image processing device, thermal image processing system, and thermal image processing method relating to embodiments of the present disclosure are described in detail with reference to the drawings.

Embodiment 1.
FIG. 1 is a diagram showing an example of the configuration of a thermal image processing system according to the first embodiment. The thermal image processing system of this embodiment includes a thermal sensor 100 that detects heat, a thermal image generating unit 200 that is a thermal image processing device that generates a thermal image based on the heat detected by the thermal sensor 100, and a thermal image display device 300 that displays the thermal image generated by the thermal image generating unit 200. The thermal image generating unit 200 includes a thermal image acquiring unit 210 that sequentially acquires the heat detected by the thermal sensor 100 as a thermal image and temporarily stores it, a thermal image storage unit 220 that stores the thermal images in the order of acquisition, i.e., in chronological order, a privacy area calculating unit 230 that calculates a privacy area that is an area including privacy information related to the living space, and a modified thermal image generating unit 240 that partially modifies and regenerates the thermal image. The privacy information related to the living space is information indicating the arrangement of a sofa or bed, the appearance of the room, etc. The modified thermal image generating unit 240 includes a thermal image modifying unit 241 that modifies the thermal image, and a modified area information holding unit 242 that stores modified area information of the privacy area.

The thermal image processing device of this embodiment is mounted on an electronic device such as an air conditioner 1. FIG. 2 is a diagram showing an example of an air conditioner 1 equipped with a thermal image processing device according to the first embodiment. In the example shown in FIG. 2, the thermal sensor 100 and the thermal image generating unit 200 are provided in the air conditioner 1, and the thermal image display device 300 is realized by installing application software on a smartphone. Note that FIG. 2 is only an example, and the method of mounting the thermal image processing device of this embodiment on the air conditioner 1 is not limited to the example shown in FIG. 2. Furthermore, the thermal image processing device of this embodiment may be mounted on an electronic device other than the air conditioner 1. An example of an electronic device other than the air conditioner 1 is a thermal detector.

The thermal image acquisition unit 210 in FIG. 1 will be described in detail. FIG. 3 is a flowchart showing an example of a data processing method of the thermal image acquisition unit 210 according to the first embodiment. Hereinafter, the period during which the thermal image acquisition unit 210 acquires one thermal image will be referred to as the thermal image acquisition period r. Also, n is an integer indicating the number of times the thermal image acquired by the thermal image acquisition unit 210 has acquired the thermal image. The initial value of n is 1.

The thermal image acquisition unit 210 receives the thermal image acquired for the nth time from the thermal sensor 100 (hereinafter referred to as the nth thermal image) (step S21). The thermal image acquisition unit 210 temporarily stores the nth thermal image received in step S21 (step S22). The thermal image acquisition unit 210 transmits the nth thermal image to the thermal image storage unit 220 (step S23a). As a step parallel to step S23a, the thermal image acquisition unit 210 transmits the nth thermal image to the corrected thermal image generation unit 240 (step S23b). The thermal image acquisition unit 210 discards the nth thermal image that was temporarily stored (step S24). n is updated to a value of n+1 (step S25), and the process from step S21 is repeated.

The thermal image storage unit 220 in Fig. 1 will be described in detail. Fig. 4 is a flowchart showing an example of a data processing method of the thermal image storage unit 220 according to the first embodiment. The thermal image storage unit 220 receives the n-th thermal image transmitted from the thermal image acquisition unit 210 in step S23a described above (step S31). The thermal image storage unit 220 stores the n-th thermal image received from the thermal image acquisition unit 210 and the thermal images already stored in chronological order, which is the order of reception (step S32).

Figure 5 is a conceptual diagram of thermal images accumulated in chronological order in the thermal image accumulation unit 220 according to the first embodiment. As shown in Figure 5, in step S32, the thermal image accumulation unit 220 stores the thermal images in chronological order starting from the thermal image of n=1. The thermal images stored in step S32 are called time-series accumulated thermal images. The time from when the thermal image accumulation unit 220 starts acquiring the thermal image of n=1 to when it finishes acquiring the nth thermal image is defined as the thermal image acquisition accumulated time t. The thermal image acquisition accumulated time t can be expressed as t = r x n.

Next, the thermal image storage unit 220 determines whether the thermal image acquisition cumulative time t is equal to or greater than a predetermined time length T (step S33). If the thermal image acquisition cumulative time t is less than the time length T (No in step S33), n is updated to n+1 (step S34), and the processes from step S31 to step S34 are repeated.

If the thermal image acquisition cumulative time t is equal to or greater than the time length T (step S33 Yes), the thermal image storage unit 220 transmits the time-series cumulative thermal image to the privacy area calculation unit 230 (step S35). When the thermal image acquisition cumulative time t is equal to or greater than the time length T, the thermal image storage unit 220 initializes the value of n to 1. In this way, when k = T/r, the time-series cumulative thermal images, which are the first to kth thermal images, are transmitted to the privacy area calculation unit 230. At this time, the time length T is, for example, about several days, and multiple setting values such as 3 days for simple setting, 4 days for standard setting, and 5 days for accurate setting may be set as an example of the setting value of the time length T, and the user may be able to select from the multiple setting values.

The privacy area calculation unit 230 in FIG. 1 will be described in detail. The privacy area calculation unit 230 measures the threshold overtime, which is the time during which the pixel value continuously exceeds the temperature threshold, for each pixel of a plurality of thermal images captured at the same location at different times, and counts the threshold overtime count, which indicates the number of times the threshold overtime is measured within a certain period of time, and determines whether the pixel is a pixel that requires correction using the threshold overtime and the threshold overtime count. FIG. 6 is a flowchart showing an example of a data processing method of the privacy area calculation unit 230 according to the first embodiment. The privacy area calculation unit 230 receives the first to kth time-series cumulative thermal images transmitted from the thermal image storage unit 220 in step S35 (step S41).

Next, the privacy area calculation unit 230 measures the threshold exceeding time and the threshold exceeding count for each pixel of the time-series cumulative thermal image received from the thermal image storage unit 220 (step S42). Using the measurement results of step S42, the privacy area calculation unit 230 calculates correction area information indicating whether or not each pixel needs to be corrected (step S43). Details of the method of measuring the threshold exceeding time and the threshold exceeding count (step S42) and the method of calculating the correction area information (step S43) shown in FIG. 6 will be described later. The privacy area calculation unit 230 stores the correction area information calculated in step S43 in a flag memory (described later) of the correction area information storage unit 242 of the correction thermal image generation unit 240 (step S44).

The method for measuring the threshold exceeding time and the threshold exceeding count in step S42 will now be described in detail. Figure 7 is a graph showing an example of temperature change when the temperature value of an arbitrary pixel (x, y) in a thermal image is plotted over time. The temperature threshold θ shown in Figure 7 is a temperature threshold at which the privacy area calculation unit 230 determines whether an arbitrary pixel (x, y) belongs to an area where a heat source of a human or other animal is present.

The privacy area calculation unit 230 compares the temperature of the pixel (x, y) of the first thermal image among the time-series accumulated thermal images with the temperature threshold θ, and starts measuring the elapsed time with a timer when the temperature exceeds the temperature threshold θ. The privacy area calculation unit 230 sequentially compares the temperature of the pixel (x, y) with the temperature threshold θ for each of the second, third, ..., kth thermal images, and starts measuring the timer if the temperature in the previous comparison was equal to or less than the temperature threshold θ and exceeds the temperature threshold θ this time, and continues measuring the timer while the temperature continues to exceed the temperature threshold θ.

On the other hand, the privacy area calculation unit 230 stops the timer measurement when the temperature of pixel (x, y) falls below the temperature threshold θ. The time indicated by the timer from the start to the stop of the timer measurement is the threshold exceeding time τ. The privacy area calculation unit 230 resets the timer measurement each time the process of stopping the timer measurement of pixel (x, y) is performed. The initial value of the threshold exceeding time τ is 0.

In addition, if the threshold exceedance time τ has been measured one or more times, i.e., if there is a thermal image in which the temperature of pixel (x, y) exceeds the threshold, the privacy area calculation unit 230 measures the number of times the threshold exceedance time τ has been measured, i.e., the number of times the temperature threshold θ has been exceeded. Specifically, the privacy area calculation unit 230 adds 1 to the value of the number of times the temperature threshold θ has been exceeded for each process from the start to the stop of measurement of the threshold exceedance time τ of pixel (x, y). The number of times the temperature threshold θ has been exceeded in one time-series accumulated thermal image is the threshold exceedance number N. The initial value of N is 1.

The privacy area calculation unit 230 measures the threshold exceedance time τ and the threshold exceedance number N for all pixels of the time-series cumulative thermal image as described above. In this manner, the privacy area calculation unit 230 measures the threshold exceedance time τ and the threshold exceedance number N for all pixels of the time-series cumulative thermal image.

Figure 8 is a flowchart showing an example of a method for calculating correction area information by the privacy area calculation unit 230 according to the first embodiment. The method for calculating correction area information described in step S43 will be described in detail with reference to the flowchart in Figure 8. In this embodiment, a pixel determined by the privacy area calculation unit 230 not to require correction is called a non-correction pixel, and a pixel determined by the privacy area calculation unit 230 to require correction is called a correction pixel. The correction area information holding unit 242 possesses a flag memory corresponding to each pixel (x, y) of the thermal image, and the value stored as a flag in the flag memory becomes the correction area information.

The privacy area calculation unit 230 selects a pixel to be processed and determines whether the threshold exceedance count N corresponding to the pixel to be processed is less than the count threshold M (step S51). The count threshold M is a threshold related to the threshold exceedance count N, and is a threshold used by the privacy area calculation unit 230 to determine whether the pixel to be processed is a modified pixel belonging to an area where people or other animals rarely appear.

If the threshold exceedance count N is less than the count threshold M (step S51 Yes), the privacy area calculation unit 230 determines that the pixel to be processed is an unmodified pixel belonging to an area where humans or other animals rarely appear. In this embodiment, each flag in the modified area information is a binary value of 0 or 1, and if the flag value (hereinafter referred to as the flag value) is 0, it indicates that the corresponding pixel is a modified pixel, and if the flag value is 1, it indicates that the corresponding pixel is an unmodified pixel. The privacy area calculation unit 230 stores a value of 1 as a flag in the flag memory corresponding to the pixel (x, y) to be processed in the modified area information storage unit 242 (step S52), and ends the calculation process of the modified area information.

If the threshold exceeding count N is equal to or greater than the count threshold M (step S51 No), the privacy area calculation unit 230 determines whether the threshold exceeding time τ of the pixel to be processed is greater than the time lower limit threshold β and less than the time upper limit threshold α (step S53). The time lower limit threshold β and the time upper limit threshold α are thresholds related to the threshold exceeding time τ. The time upper limit threshold α is a threshold for determining whether the pixel to be processed is an unmodified pixel belonging to an area where a heat source other than a person due to a home appliance or sunlight exists, and the time lower limit threshold β is a threshold for distinguishing between a short temperature change including air fluctuation and the appearance of a person or other animal. It is also possible to set the time lower limit threshold β and the time upper limit threshold α according to the time or season in which the user uses electronic devices including the thermal image processing device.

If the threshold exceeding time τ is greater than the time lower limit threshold β and less than the time upper limit threshold α (step S53 Yes), the privacy area calculation unit 230 determines that the pixel to be processed is a modified pixel that belongs to an area where people frequently appear. If the threshold exceeding time τ is measured multiple times during the time length T, the average, median, maximum value, etc. of the multiple measured threshold exceeding times τ can be used as the threshold exceeding time τ used in the determination of step S53. The privacy area calculation unit 230 stores a value of 0 as a flag in the flag memory corresponding to the pixel to be processed in the modified area information storage unit 242 (step S54), and ends the calculation process of the modified area information.

If the threshold exceeding time τ is equal to or greater than the upper time threshold α or equal to or less than the lower time threshold β (step S53 No), the privacy area calculation unit 230 stores a value of 1 as a flag in the flag memory corresponding to the pixel (x, y) to be processed in the correction area information storage unit 242 (step S55), and ends the calculation process of the correction area information. If the threshold exceeding time τ is equal to or greater than the upper time threshold α, the pixel (x, y) to be processed is considered to belong to an area where a heat source due to an electrical appliance or sunlight exists, and if the threshold exceeding time τ is equal to or less than the lower time threshold β, it is considered to be a short temperature change including air fluctuations. Thus, if the result of step S53 is No, the pixel (x, y) to be processed is considered to be a non-corrected pixel.

Steps S51 to S55 are repeated until the calculation of the modified area information for all pixels of the time-series accumulated thermal image and the storage in the flag memory of the modified area information storage unit 242 are completed. In this manner, the privacy area calculation unit 230 generates modified area information corresponding to the thermal image received from the thermal image storage unit 220 and stores it in the modified area information storage unit 242.

The generation of this correction area information is performed, for example, for each thermal image of time length T, and the correction area information is updated in the correction area information storage unit 242 for each time length T. Alternatively, the correction area information may be generated periodically for a period longer than the time length T, such as one day or several days.

In the above example, both the measured values of the number of times the threshold is exceeded N and the time it takes to exceed the threshold τ are used, but it is also possible to use only the number of times the threshold is exceeded N to determine whether the pixel (x, y) being processed is a non-modified pixel or a modified pixel. When using only the number of times the threshold is exceeded N, there is a method in which the appearance and disappearance of the heat source are counted as one occurrence, and a method in which a count value is added each time a heat source is detected independently for each frame in which a thermal image is acquired. Either method may be used, but in either case, if the number of times the threshold is exceeded N is greater than the lower threshold and less than the upper threshold, it is determined to be a modified pixel, and if the number of times the threshold is exceeded N is less than the lower threshold or greater than the upper threshold, it is determined to be a non-modified pixel, thereby generating modified area information.

In the case of the former method, the value of the number of times N that the threshold is exceeded when a heat source from electronic devices, including home appliances, is detected for a long time is small, and the value of the number of times N that the threshold is exceeded when a short temperature change including air fluctuation is detected is large. For this reason, in the case of the former method, the lower limit threshold is set to a value that can distinguish between a case where a heat source from electronic devices is detected for a long time and the appearance of a person or other animal, and the upper limit threshold is set to a value that can distinguish between a short fluctuation such as air fluctuation and the appearance of a person or other animal. In the case of the latter method, the value of the number of times N that the threshold is exceeded when a heat source from electronic devices, including home appliances, is detected for a long time is large, and the value of the number of times N that the threshold is exceeded when a short temperature change including air fluctuation is detected is small. For this reason, in the case of the latter method, the lower limit threshold is set to a value that can distinguish between a short fluctuation such as air fluctuation and the appearance of a person or other animal, and the upper limit threshold is set to a value that can distinguish between a case where a heat source from electronic devices is detected for a long time and the appearance of a person or other animal.

The modified thermal image generating unit 240 in FIG. 1 will be described in detail. The modified thermal image generating unit 240 modifies the thermal image using the result of the judgment by the privacy area calculating unit 230, i.e., the flag indicating, for each pixel, whether the pixel is a pixel that requires correction, with a binary value of 0 or 1. More specifically, the modified thermal image generating unit 240 modifies the thermal image using the correction area information indicating the result of the judgment regarding all pixels of the thermal image. FIG. 9 is a flowchart showing an example of a data processing method of the thermal image of the modified thermal image generating unit 240 according to the first embodiment. The thermal image correction unit 241 of the modified thermal image generating unit 240 receives the n-th thermal image from the thermal image acquiring unit 210 (step S61). Step S61 is a process of receiving the n-th thermal image transmitted by the thermal image acquiring unit 210 in step S23b.

The thermal image correction unit 241 corrects the n-th thermal image using the correction area information stored in the correction area information storage unit 242 (step S62). The thermal image corrected by the corrected thermal image generation unit 240 in step S62 (hereinafter referred to as the corrected thermal image) is transmitted to the thermal image display device 300 (step S63).

The thermal image correction processing method in step S62 will be described in detail. The thermal image correction unit 241 selects an arbitrary pixel (i, j) and sets it as the pixel (i, j) to be processed. FIG. 10 is a conceptual diagram showing an example of correction area information calculated by the privacy area calculation unit 230 according to the first embodiment. As shown in FIG. 10, the thermal image correction unit 241 sets the sum of flag values s ij of the processing object pixel (i, j) located at the i-th position in the horizontal direction and the j-th position in the vertical direction of the thermal image and its surrounding pixels as the total value S. The flag value s _ij in this embodiment is the value of the flag corresponding to each pixel in _the correction area information.

The thermal image correction unit 241 calculates a total value S of the flag value s _ij of the pixel (i, j) to be processed, which is located at the i-th position in the horizontal direction and the j-th position in the vertical direction of the thermal image, and the flag values s _ij of the pixels surrounding the pixel (i, j). An example of a formula for calculating the total value S is shown in Formula (1). In the calculation example of Formula (1), the total value S is the total of the flag values of the pixel (i, j) to be processed and the surrounding pixels adjacent to the pixel (i, j) to be processed.

The threshold value of the sum value S for the thermal image correction unit 241 to determine whether or not pixel correction is necessary is set as a sum threshold value Z. If the sum value S is less than the sum threshold value Z, the thermal image correction unit 241 corrects the processing target pixel (i, j) according to, for example, the following formula (2). The pixel value _I'ij indicates the corrected pixel value of the i-th pixel in the horizontal direction and the j-th pixel in the vertical direction of the thermal image.

FIG. 11 is a conceptual diagram showing an example of a thermal image correction processing method by the thermal image correction unit 241 according to the first embodiment. The left diagram of FIG. 11 shows a thermal image before correction, and the right diagram of FIG. 11 shows a corrected thermal image, which is a thermal image after correction. When the total value S is less than the total threshold value Z, the thermal image correction unit 241 calculates the pixel value of the pixel hatched with diagonal lines in the right diagram by applying an averaging filter to, for example, the 9 pixels of 3×3 at the upper right corner of the left diagram. However, the filter used to obtain the corrected pixel value _I′ij includes a smoothing filter such as a Gaussian filter in addition to the averaging filter. In addition, the filter size can be set to any size other than 3×3.

In this way, the thermal image correction unit 241 calculates the sum of the flag values corresponding to each pixel and its adjacent pixels, and when the sum is equal to or greater than a threshold, corrects the thermal image by smoothing filtering the pixel values of the pixel and its adjacent pixels. In addition to filtering, other correction methods include a method of blurring pixels determined to require correction by superimposing data used for special processing that has been set in advance.

In this way, the privacy region calculation unit 230 stores the result of the determination indicating whether or not correction is required for each pixel in the correction region information storage unit 242 as correction region information, and the correction region information storage unit 242 stores the correction region information before updating until the correction region information is updated by the privacy region calculation unit 230. Then, the corrected thermal image generation unit 240 corrects the thermal images acquired sequentially using the correction region information stored by the correction region information storage unit 242. Therefore, it is not necessary to determine whether or not an image is required each time an image is acquired, and the processing load can be reduced.

The thermal image display device 300 in FIG. 1 will be described in detail. In step S63, the thermal image display device 300 receives the modified thermal image transmitted by the modified thermal image generation unit 240. The thermal image display device 300 displays the received modified thermal image on a monitor or display. In this manner, the thermal image display device 300 displays the modified thermal image generated by the thermal image generation unit 200.

Examples of the thermal image display device 300 include electronic terminals including smartphones and monitors. The functions of the thermal image display device 300 are realized by a processing circuit and a display means such as a display or monitor. The processing circuit may be dedicated hardware or a control circuit equipped with a processor such as a CPU (also called a Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)).

Here, the hardware configuration of the thermal image generating unit 200 of this embodiment will be described. The functions of each part of the thermal image generating unit 200 are realized by the processing circuit 400. The processing circuit 400 may be dedicated hardware or a control circuit equipped with a processor such as a CPU.

When the processing circuit 400 is a dedicated hardware, the processing circuit 400 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. The functions of each part of the thermal image generating unit 200 may be realized by different processing circuits 400, or the functions of each part may be realized by a single processing circuit 400.

Figure 12 is a diagram showing an example of the configuration of the processing circuit 400 of this embodiment. The processing circuit 400 includes a processor 401 and a memory 402. When the functions of each part of the thermal image generating unit 200 and some of the functions of the thermal image display device 300 are realized by the processing circuit 400, these functions are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 402. The functions of each part are realized by the processor 401 reading and executing the programs stored in the memory 402. That is, the processing circuit 400 includes a memory 402 for storing a program that results in the execution of the steps of acquiring a thermal image, accumulating information about the thermal image in chronological order, measuring the threshold overage time τ, which is the time that the pixel value continuously exceeds the temperature threshold θ, for each pixel of multiple thermal images captured at the same location at different times, and counting the threshold overage number N, which indicates the number of times the threshold overage time τ is measured within a certain period of time, and determining whether or not the pixel in question is a pixel that requires correction using the threshold overage time τ and the threshold overage number N, and a correction step of correcting the thermal image using the result of the determination.

These programs can also be said to cause the computer to execute the functions of each part of the thermal image generating unit 200 and the procedures or methods in the thermal image display device 300. Here, memory refers to, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (registered trademark) (Electrically Erasable Programmable Read Only Memory), as well as magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs (Digital Versatile Discs).

In addition, the functions of each part of the thermal image generating unit 200 and some of the functions of the thermal image display device 300 may be partially realized by dedicated hardware and partially realized by software or firmware.

As described above, by defining a place where a person or other animal stays for a relatively long time as a privacy area, it is possible to protect privacy information related to an individual's living space, such as a sofa or a bed. In addition, by calculating the modified area information indicating the privacy area related to the living space once and storing it in the modified area information storage unit 242, it becomes unnecessary to apply image processing techniques that have a relatively high computational load, such as face detection, and it is possible to process thermal images efficiently.

Embodiment 2.
In the above-described first embodiment, as shown in Fig. 1, the thermal image generating unit 200 is configured to process thermal images using hardware mounted on the air conditioner 1, but a second embodiment will be described in which the thermal image generating unit 200 processes thermal images in the cloud. Components having the same functions as those in the first embodiment are given the same reference numerals as those in the first embodiment, and duplicated explanations will be omitted. Below, differences from the first embodiment will be mainly described.

13 is a diagram showing an example of the configuration of a thermal image processing system in which the thermal image storage unit 220, the privacy area calculation unit 230, and the modified thermal image generation unit 240 according to the second embodiment are implemented in the cloud 2. The cloud 2 is composed of, for example, a plurality of computer systems. The cloud 2 and the air conditioner 1 may be connected by wire or wirelessly (the same applies to the following figures). Each computer system includes a processor 401 and a memory 402, similar to the processing circuit 400 shown in FIG. 12. The processor 401 reads and executes a program stored in the memory 402, thereby realizing the functions of each unit. In this embodiment, the thermal sensor 100 mounted on the air conditioner 1 transmits a thermal image to the thermal image acquisition unit 210. Next, the thermal image storage unit 220 transmits a time-series cumulative thermal image composed of a plurality of thermal images corresponding to the time length T to the privacy area calculation unit 230 on the cloud 2 via a wide area network such as the Internet. The privacy region calculation unit 230 calculates correction region information on the cloud 2 and stores it in a correction region information storage unit 242 (not shown in FIG. 13) of the correction thermal image generation unit 240 .

The thermal image correction unit 241 (not shown in FIG. 13) of the corrected thermal image generation unit 240 applies image processing to the thermal image on the cloud 2 using the correction area information stored in the correction area information storage unit 242 to create a corrected thermal image. The corrected thermal image generation unit 240 transmits the corrected thermal image to the thermal image display device 300 via a wide area network such as the Internet, and the thermal image display device 300 displays the corrected thermal image on a display or monitor.

13, the thermal image processing system includes a thermal sensor 100 for acquiring a thermal image, an air conditioner 1 connected to a cloud 2, which is a computer system, via a wide area network such as the Internet, and a thermal image display device 300, which is a terminal. The air conditioner 1 transmits a thermal image to the thermal image processing device, and the cloud 2 measures the threshold overtime τ, which is the time during which the pixel value continuously exceeds the temperature threshold θ, for each pixel of the captured thermal images, and counts the threshold overtime N, which indicates the number of times the threshold overtime τ is measured within a certain period of time, and uses the threshold overtime τ and the threshold overtime N to determine whether the pixel is a pixel that requires correction. The cloud 2 then uses the result of the determination to correct the thermal image, transmits the corrected thermal image to the thermal image display device 300, and the thermal image display device 300 displays the corrected thermal image.

In the configuration of Figure 13, the thermal image storage unit 220, privacy area calculation unit 230, and corrected thermal image generation unit 240 are implemented in cloud 2, and the main thermal image correction processing is concentrated in cloud 2, thereby achieving the effect of reducing the calculation load in the air conditioner 1 and the thermal image display device 300.

Figure 14 is a diagram showing an example configuration of a thermal image processing system in which the thermal image storage unit 220 and privacy area calculation unit 230 according to embodiment 2 are implemented in cloud 2, and the modified thermal image generation unit 240 is implemented in a thermal image display device 300. The thermal sensor 100 mounted on the air conditioner 1 sequentially transmits the acquired thermal images to the thermal image acquisition unit 210. The thermal image acquisition unit 210 transmits the acquired thermal images to the thermal image storage unit 220 on cloud 2 via a wide area network such as the Internet.

Next, the thermal image storage unit 220 transmits a time-series accumulated thermal image composed of a plurality of thermal images corresponding to the time length T to the privacy area calculation unit 230 on the cloud 2. The privacy area calculation unit 230 calculates correction area information on the cloud 2 and transmits the correction area information to the thermal image display device 300. The thermal image display device 300, which is a terminal such as a smartphone, has the functions of the corrected thermal image generation unit 240 implemented as application software.

The thermal image correction unit 241 (not shown in FIG. 14) of the corrected thermal image generating unit 240 implemented in the thermal image display device 300 receives the thermal image from the thermal image acquiring unit 210 via a wide area network such as the Internet. The correction area information holding unit 242 (not shown in FIG. 14) of the corrected thermal image generating unit 240 stores the correction area information received from the privacy area calculating unit 230 via a wide area network such as the Internet. The thermal image correction unit 241 performs image processing using the correction area information on the electronic terminal of the thermal image display device 300 to create a corrected thermal image. The corrected thermal image generating unit 240 transmits information of the corrected thermal image via the processing circuit 400 of the thermal image display device 300, and the thermal image display device 300 displays the corrected thermal image on a display or monitor.

Thus, the thermal image processing system of the example shown in FIG. 14 includes a thermal sensor 100 for acquiring a thermal image, an air conditioner 1 connected to a cloud 2, which is a computer system, via a wide area network such as the Internet, and a thermal image display device 300, which is a terminal. The air conditioner 1 transmits a thermal image to the cloud 2 and the thermal image display device 300. The cloud 2 measures the threshold overtime τ, which is the time during which the pixel value continuously exceeds the temperature threshold θ, for each pixel of the captured thermal images, and counts the threshold overtime N, which indicates the number of times the threshold overtime τ is measured within a certain period of time. The cloud 2 uses the threshold overtime τ and the threshold overtime N to determine whether the pixel is a pixel that requires correction, and transmits the result of the determination to the thermal image display device 300. The thermal image display device 300 uses the result of the determination to correct the received thermal image and display the corrected thermal image.

In the configuration of Figure 14, by implementing the corrected thermal image generation unit 240 in the thermal image display device 300, the thermal image transmitted directly from the air conditioner 1 via a wide area network such as the Internet is corrected by the corrected thermal image generation unit 240 on the thermal image display device 300 using the correction area information calculated on the cloud 2, thereby reducing the amount of communication.

Figure 15 is a diagram showing an example configuration of a thermal image processing system in which the thermal image storage unit 220 and privacy area calculation unit 230 according to embodiment 2 are implemented in the cloud 2, and the modified thermal image generation unit 240 is implemented in the air conditioner 1. The thermal image acquisition unit 210 sequentially transmits thermal images received from the thermal sensor 100 mounted on the air conditioner 1 to the thermal image storage unit 220 via a wide area network such as the Internet. As parallel processing, the thermal image acquisition unit 210 transmits the thermal image received from the thermal sensor 100 to the modified thermal image generation unit 240 implemented in the air conditioner 1.

Next, the thermal image storage unit 220 transmits a time-series accumulated thermal image composed of multiple thermal images corresponding to the time length T to the privacy area calculation unit 230 via the cloud 2. The privacy area calculation unit 230 transmits the modified area information calculated on the cloud 2 to the air conditioner 1, and the air conditioner 1 stores the modified area information in the modified thermal image generation unit 240. The modified thermal image created by the modified thermal image generation unit 240 is transmitted to the thermal image display device 300 via a wide area network such as the Internet, and the thermal image display device 300 displays the modified thermal image on a display or monitor.

Thus, the thermal image processing system of the example shown in FIG. 15 includes a thermal sensor 100 for acquiring a thermal image, an air conditioner 1 connected to a cloud 2, which is a computer system, via a wide area network such as the Internet, and a thermal image display device 300, which is a terminal. The air conditioner 1 transmits a thermal image to the cloud 2, and the cloud 2 measures the threshold overtime τ, which is the time during which the pixel value continuously exceeds the temperature threshold θ, for each pixel of the captured thermal images, and counts the threshold overtime count N, which indicates the number of times the threshold overtime τ is measured within a certain period of time, and judges whether the pixel in question is a pixel that requires correction using the threshold overtime τ and the threshold overtime count N, and transmits the result of the judgment to the air conditioner 1. The air conditioner 1 uses the result of the judgment to correct the thermal image, and transmits the corrected thermal image to the thermal image display device 300. The thermal image display device 300 displays the corrected thermal image received from the air conditioner 1.

In the configuration of FIG. 15, unlike the configuration of FIG. 14, the corrected thermal image generation unit 240 is implemented in the air conditioner 1, and the thermal image is corrected within the air conditioner 1 and then transmitted to the thermal image display device 300 via a wide area network such as the Internet. When transmitting the uncorrected thermal image to the thermal image display device 300 via a wide area network such as the Internet as in the configuration of FIG. 14, there is a possibility that the uncorrected thermal image will be leaked if there is unauthorized access to the cloud 2. However, in the configuration of FIG. 15, the thermal image is corrected within the air conditioner 1 and then transmitted to the thermal image display device 300 via a wide area network such as the Internet, so that the possibility of the uncorrected thermal image being leaked on the line between the air conditioner 1 and the terminal is extremely low, and the effect of privacy protection can be enhanced.

In the configurations of Figures 13 to 15, by implementing in cloud 2 a thermal image storage unit 220 that stores time-series accumulated thermal images, a privacy area calculation unit 230 that calculates correction area information for the time-series accumulated thermal images, and a corrected thermal image generation unit 240 that corrects the thermal image using the correction area information, it is possible to reduce the load on electronic terminals with limited computational resources.

The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or the embodiments may be combined with each other. Also, parts of the configurations may be omitted or modified without departing from the spirit of the invention.

1 Air conditioner, 2 Cloud, 100 Thermal sensor, 200 Thermal image generation unit, 210 Thermal image acquisition unit, 220 Thermal image storage unit, 230 Privacy area calculation unit, 240 Corrected thermal image generation unit, 241 Thermal image correction unit, 242 Corrected area information storage unit, 300 Thermal image display device, 400 Processing circuit, 401 Processor, 402 Memory.

Claims (8)

a privacy area calculation unit that counts the number of times that a pixel value exceeds a temperature threshold within a certain period of time for each pixel of a plurality of thermal images captured at the same location at different times, and determines whether or not the pixel is a pixel that requires correction using the number of times that the pixel value exceeds the threshold;
a corrected thermal image generating unit that corrects the thermal image using the result of the determination by the privacy region calculating unit;
Equipped with
the result of the determination is a flag for each of the pixels, which indicates, with a binary value of 0 or 1, whether or not the pixel is a pixel that requires correction;
the corrected thermal image generating unit includes a corrected area information storage unit that stores corrected area information indicating the result of the determination regarding all pixels of the thermal image;
The privacy region calculation unit stores a result of the determination as the correction region information in the correction region information storage unit,
The correction area information holding unit holds the correction area information before updating until the correction area information is updated by the privacy area calculation unit,
The corrected thermal image generating unit corrects the thermal images acquired sequentially using the correction area information stored in the correction area information storage unit.
Thermal imaging device. 2. The thermal image processing device according to claim 1, wherein the privacy area calculation unit measures a threshold overtime, which is the time during which a pixel value continuously exceeds a temperature threshold for each pixel of the plurality of thermal images, counts the number of times the threshold overtime is measured within the certain period of time as the threshold overtime, and determines whether or not the pixel is a pixel that requires correction using the threshold overtime and the threshold overtime. the corrected thermal image generating unit includes a thermal image correcting unit that reads the corrected area information from the corrected area information storage unit and corrects the thermal image using the corrected area information;
The thermal image processing device according to claim 1 or 2, wherein the thermal image correction unit calculates, for each pixel, the sum of the values of the flags corresponding to that pixel and pixels adjacent to that pixel, and when the sum is greater than or equal to a threshold value, corrects the thermal image by smoothing filtering the pixel values of that pixel and pixels adjacent to that pixel. An air conditioner comprising the thermal image processing device according to any one of claims 1 to 3 ;
A terminal,
Equipped with
The air conditioner transmits the corrected thermal image to the terminal;
The terminal displays the corrected thermal image received from the air conditioner. an air conditioner including a thermal sensor for acquiring a thermal image and connected to a computer system via a wide area network;
A terminal,
Equipped with
The air conditioner transmits the thermal image to the computer system;
the computer system counts a threshold exceedance count, which indicates the number of times that a pixel value exceeds a temperature threshold within a certain period of time, for each pixel of the captured thermal images, determines whether or not the pixel is a pixel that requires correction using the threshold exceedance count, corrects the thermal image using a result of the determination, and transmits the corrected thermal image to the terminal;
The terminal displays the corrected thermal image received from the air conditioner;
the result of the determination is a flag for each of the pixels, which indicates, with a binary value of 0 or 1, whether or not the pixel is a pixel that requires correction;
the computer system includes a modified area information storage unit that stores modified area information indicating the result of the determination regarding all pixels of the thermal image;
the computer system stores a result of the determination as the modified area information in the modified area information storage unit;
the correction area information storage unit stores the correction area information before the update until the correction area information is updated;
The computer system corrects the thermal images acquired sequentially using the correction area information stored in the correction area information storage unit.
Thermal imaging system. an air conditioner including a thermal sensor for acquiring a thermal image and connected to a computer system via a wide area network;
A terminal,
Equipped with
The air conditioner transmits the thermal image to the computer system and the terminal;
the computer system counts a threshold exceedance count, which indicates the number of times that a pixel value exceeds a temperature threshold within a certain period of time, for each pixel of the captured thermal images, and determines whether or not the pixel is a pixel that requires correction using the threshold exceedance count, and transmits the result of the determination to the terminal;
The terminal modifies the thermal image received from the air conditioner using the result of the determination, and displays the modified thermal image ;
the result of the determination is a flag for each of the pixels, which indicates, with a binary value of 0 or 1, whether or not the pixel is a pixel that requires correction;
The terminal includes a correction area information storage unit that stores correction area information indicating the result of the determination regarding all pixels of the thermal image,
The terminal stores a result of the determination as the correction area information in the correction area information storage unit,
the correction area information storage unit stores the correction area information before the update until the correction area information is updated;
The terminal corrects the thermal images acquired sequentially using the correction area information stored in the correction area information storage unit.
Thermal imaging system. an air conditioner including a thermal sensor for acquiring a thermal image and connected to a computer system via a wide area network;
A terminal,
Equipped with
The air conditioner transmits the thermal image to the computer system;
The computer system counts a threshold exceedance count, which indicates the number of times that a pixel value exceeds a temperature threshold within a certain period of time, for each pixel of the captured thermal images, and determines whether or not the pixel is a pixel that requires correction using the threshold exceedance count, and transmits the result of the determination to the air conditioner.
The air conditioner modifies the thermal image using a result of the determination, and transmits the modified thermal image to the terminal;
The terminal displays the corrected thermal image received from the air conditioner ;
the result of the determination is a flag for each of the pixels, which indicates, with a binary value of 0 or 1, whether or not the pixel is a pixel that requires correction;
The air conditioner includes a correction area information storage unit that stores correction area information indicating the result of the determination regarding all pixels of the thermal image,
The air conditioner stores a result of the determination as the correction area information in the correction area information storage unit,
the correction area information storage unit stores the correction area information before the update until the correction area information is updated;
The air conditioner corrects the thermal images acquired sequentially using the correction area information stored in the correction area information storage unit.
Thermal imaging system. A thermal image processing method in a thermal image processing device, comprising:
a determination step of counting the number of times that a pixel value exceeds a temperature threshold within a certain time period for each pixel of a plurality of thermal images captured at the same location at different times, and determining whether or not the pixel is a pixel that requires correction using the number of times that the pixel value exceeds the threshold;
a correction step of correcting the thermal image using a result of the determination;
Including,
the result of the determination is a flag for each of the pixels, which indicates, with a binary value of 0 or 1, whether or not the pixel is a pixel that requires correction;
The thermal image processing device includes a correction area information storage unit that stores correction area information indicating the result of the determination regarding all pixels of the thermal image,
In the determination step, the thermal image processing device stores a result of the determination as the correction area information in the correction area information storage unit,
the correction area information storage unit stores the correction area information before the update until the correction area information is updated;
In the correcting step, the thermal image processing device corrects the thermal images acquired sequentially using the correction area information stored in the correction area information storage unit.
Thermal imaging methods.

Images