JP6797344B1 - Learning device, utilization device, program, learning method and utilization method - Google Patents

Abstract

Includes a thermal image that images the temperature distribution of the subject by using infrared rays emitted from the subject, and a visible image that images the subject by using visible light reflected from the subject. A trained model for inferring a visible image from a thermal image by learning inference from a thermal image to a visible image using the learning side data acquisition unit (112) that acquires training data and the training data. It is characterized by including a model generation unit (113) to be generated.

Description

The present disclosure relates to learning devices, utilization devices, programs, learning methods and utilization methods.

A general thermal infrared solid-state image sensor (hereinafter referred to as a thermal image sensor) visualizes incident infrared rays emitted by a subject, and the difference in temperature rise caused by absorbing the infrared rays is the shade of the image. The infrared rays emitted by the subject are focused by the lens and imaged on the image sensor.

While a thermal image sensor capable of acquiring thermal information can acquire information that cannot be acquired by a visible camera, for example, an inexpensive small sensor has a high image resolution, contrast, contour sharpness, or SN ratio. It becomes smaller. Further, the thermal image sensor formed by the large sensor is expensive.

On the other hand, in the fields of home monitoring, smart building, crime prevention, etc., there are services that identify human behavior or posture and detect abnormal behavior. Human postures include standing (standing position), sitting (sitting position), and lying down (lying position). Thermal image sensors are advantageous because they have lower barriers to introduction than visible cameras from the viewpoint of privacy protection.

Here, a technique of generating a trained model by inputting a visible image or a distance image and a posture information (correct answer) of a subject and estimating a posture information from the visible image or a distance image using the generated trained model. (See, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2017-97577 (page 5)

Patent Document 1 describes a posture estimation device that estimates a posture from a visible image or a distance image. In this posture estimation device, there is a problem that the image quality such as the resolution or the SN ratio of the thermal image is smaller than that of the visible image or the distance image, and the posture estimation is not easy.

Therefore, one or a plurality of aspects of the present disclosure is intended to enable highly accurate estimation of the posture of a subject in a thermal image.

The learning device according to one aspect of the present disclosure uses infrared rays emitted from a subject to image a thermal image of the temperature distribution of the subject, and visible light reflected from the subject. A data acquisition unit that acquires learning data including a visible image of the subject, a contour extraction unit that extracts a contour image showing the contour of the subject from the thermal image, and the thermal image and the contour image. from the combination by learning the reasoning to the visible image, and a model generating unit for generating a learned model for inferring the visible image from the combination of the thermal image and the outline image, the learned model Is formed by a layer of a decoder portion and a layer of an encoder portion, and the decoder portion has two paths in parallel, the two paths being a path for decoding the thermal image and the contour. It is characterized in that it is a path for decoding an image .

The utilization device according to one aspect of the present disclosure utilizes infrared rays emitted from a subject to image a thermal image of the temperature distribution of the subject and visible light reflected from the subject. Using the training data including the visible image obtained by imaging the subject and the contour image data showing the contour image showing the contour of the subject extracted from the thermal image, the thermal image and the contour image are used. A storage unit that stores a trained model for inferring the visible image from the combination of the thermal image and the contour image, which is generated by learning the inference from the combination to the visible image, and a target subject. A data acquisition unit that acquires target thermal image data indicating a target thermal image that is a thermal image of a target subject, and a contour extraction unit that extracts a target contour image that is a contour image indicating the contour of the target subject from the target thermal image. And the inference unit that infers the target visible image that is the visible image of the target subject from the combination of the target thermal image and the target contour image using the trained model, and the target subject from the target visible image. The trained model is formed by a layer of a decoder portion and a layer of an encoder portion, and the decoder portion has two parallel paths. One path is a path for decoding the thermal image and a path for decoding the contour image .

The program according to one aspect of the present disclosure uses a computer to use an infrared image emitted from a subject to use a thermal image that images the temperature distribution of the subject and visible light reflected from the subject. A data acquisition unit that acquires learning data including a visible image of the subject, a contour extraction unit that extracts a contour image showing the contour of the subject from the thermal image, and the thermal image and the contour. By learning the inference from the combination of images to the visible image, it functions as a model generation unit that generates a trained model for inferring the visible image from the combination of the thermal image and the contour image, and the learning. The completed model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel, and the two paths are a path for decoding the thermal image and a path for decoding the thermal image. The path is for decoding the contour image .

The program according to one aspect of the present disclosure uses a computer to use an infrared image emitted from a subject to use a thermal image that images the temperature distribution of the subject and visible light reflected from the subject. Then, using the learning data including the visible image obtained by imaging the subject and the contour image data showing the contour image showing the contour of the subject extracted from the thermal image, the thermal image and the contour are used. A storage unit that stores a learned model for inferring the visible image from the combination of the thermal image and the contour image, which is generated by learning the inference from the combination of images to the visible image, and a target subject. A data acquisition unit that acquires target thermal image data indicating a target thermal image that is a thermal image of the target subject, and a contour extraction unit that extracts a target contour image that is a contour image indicating the contour of the target subject from the target thermal image. , The inference unit that infers the target visible image that is the visible image of the target subject from the combination of the target thermal image and the target contour image using the trained model, and the target subject from the target visible image. The trained model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel. One path is a path for decoding the thermal image and a path for decoding the contour image .

The learning method according to one aspect of the present disclosure is to use an infrared image emitted from a subject, a thermal image that images the temperature distribution of the subject, and visible light reflected from the subject. Learning data including a visible image of the subject is acquired, a contour image showing the contour of the subject is extracted from the thermal image, and inference from the combination of the thermal image and the contour image to the visible image. Is a learning method for generating a trained model for inferring the visible image from a combination of the thermal image and the contour image by learning the above, and the trained model is a layer of a decoder portion and an encoder portion. The decoder portion is formed by the layers of the above, and the decoder portion includes two paths in parallel, and the two paths are a path for decoding the thermal image and a path for decoding the contour image. It is characterized by.

In the utilization method according to the position aspect of the present disclosure, the target thermal image data showing the target thermal image which is the thermal image of the target subject which is the target subject is acquired, and the contour showing the outline of the target subject is obtained from the target thermal image. By extracting the target contour image which is an image and using the infrared rays emitted from the subject, the thermal image which imaged the temperature distribution of the subject and the visible light reflected from the subject are used. A combination of the thermal image and the contour image using learning data including a visible image obtained by imaging the subject and contour image data indicating a contour image showing the contour of the subject extracted from the thermal image. Using the trained model for inferring the visible image from the combination of the thermal image and the contour image generated by learning the inference to the visible image from the target thermal image and the target contour image. This is a utilization method in which a target visible image, which is a visible image of the target subject, is inferred from the combination of the above and the posture of the target subject is estimated from the target visible image. The trained model is a layer of a decoder portion. , The decoder portion is formed by a layer of an encoder portion, and the decoder portion has two paths in parallel, and the two paths are a path for decoding the thermal image and a path for decoding the contour image. It is characterized by being.

According to one or more aspects of the present disclosure, the posture of a subject in a thermal image can be estimated with high accuracy.

It is a block diagram which shows schematic structure of the posture estimation system which concerns on Embodiments 1 and 2. It is a block diagram which shows schematic structure of the learning apparatus in Embodiments 1 and 2. It is a schematic diagram which shows an example of a three-layer neural network. It is the schematic which shows an example of the structure of the trained model of the image conversion process which converts a thermal image into a visible image in Embodiment 1. FIG. It is a block diagram which shows the structure of a computer schematicly. It is a flowchart which shows the process which a learning apparatus learns. It is a block diagram which shows schematic structure of the posture estimation apparatus in Embodiment 1. FIG. It is a flowchart which shows the process which the attitude estimation apparatus infers the visible image corresponding to a thermal image, and estimates the attitude from the visible image. It is a block diagram which shows schematic structure of the posture estimation apparatus in Embodiment 2. FIG. It is a block diagram which shows schematic structure of the learning apparatus in Embodiment 3. FIG. It is a schematic diagram which shows an example of the structure of the trained model of the image conversion process which converts a thermal image and a contour image into a visible image in Embodiment 3. It is a block diagram which shows schematic structure of the posture estimation apparatus in Embodiment 3. FIG.

Embodiment 1.
FIG. 1 is a block diagram schematically showing the configuration of the posture estimation system 100 according to the first embodiment.
The posture estimation system 100 includes a learning device 110 that functions as a model generation device, and a posture estimation device 130 that functions as a utilization device. The processing method performed by the posture estimation device 130 is used as a utilization method.
In the posture estimation system 100, the posture estimation device 130 estimates the posture using the trained model learned by the learning device 110.

FIG. 2 is a block diagram schematically showing the configuration of the learning device 110.
The learning device 110 includes a learning side input unit 111, a learning side data acquisition unit 112, a model generation unit 113, a learning side learned model storage unit 114, and a learning side communication unit 115.

The learning side input unit 111 is an input unit that receives input of learning data. The input learning data is given to the learning side data acquisition unit 112.
Here, the learning data is teacher data showing a combination of a thermal image and a visible image as a correct answer to be inferred from the thermal image.

The thermal image is acquired by imaging the temperature distribution of the subject by using infrared rays radiated from the subject.
Further, the visible image is acquired by imaging the subject by using the visible light reflected from the subject. In the visible image, the appearance of the subject is imaged.

The learning side data acquisition unit 112 is a data acquisition unit that acquires learning data via the learning side input unit 111. The acquired learning data is given to the model generation unit 113.

The model generation unit 113 learns the visible image corresponding to the thermal image based on the learning data given from the learning side data acquisition unit 112. In other words, the model generation unit 113 generates a trained model for inferring the optimum visible image corresponding to the thermal image by learning the combination of the thermal image and the visible image shown in the training data. Specifically, the model generation unit 113 generates a trained model for inferring a visible image from a thermal image by learning inference from a thermal image to a visible image using training data.
Then, the model generation unit 113 stores the generated learned model as the learning side learned model in the learning side learned model storage unit 114.

As the learning algorithm used by the model generation unit 113, known algorithms such as supervised learning, unsupervised learning, and reinforcement learning can be used. As an example, here, a case where a neural network is applied will be described.

Here, in the case of supervised learning, the thermal image and the visible image shown in the learning data need to be paired data containing the same subject. In the case of unsupervised learning, the thermal image and the visible image do not have to contain the same subject.

The model generation unit 113 learns a visible image corresponding to a thermal image by, for example, supervised learning according to a neural network model.
Here, supervised learning is a method of learning a feature in the learning data by giving a set of input and result (label) data to the learning device as learning data, and inferring the result from the input. To say.

A neural network is composed of an input layer composed of a plurality of neurons, an intermediate layer (hidden layer) composed of a plurality of neurons, and an output layer composed of a plurality of neurons. The intermediate layer may be one layer or two or more layers.

FIG. 3 is a schematic view showing an example of a three-layer neural network.
As shown in FIG. 3, in the case of a three-layer neural network, when a plurality of input values are input to the input layers X1 to X3, the first weights w11 to w16 (hereinafter referred to as the third) are added to the input values. (Also called one weight W1) is multiplied. The calculated value, which is the value obtained by multiplying the input value by the first weights w11 to w16, is input to the intermediate layers Y1 and Y2. The calculated value is multiplied by the second weights w21 to w26 (hereinafter, also referred to as the second weight W2), and the output value obtained by multiplying the calculated value by the second weights w21 to w26 is the output layer. It is output from Z1 to Z3. This output value changes depending on the value of the first weight W1 and the value of the second weight W2.

In the present embodiment, the neural network is subjected to so-called supervised learning according to the learning data created based on the combination of the thermal image and the visible image represented by the learning data acquired by the learning side data acquisition unit 112. Learn a trained model to infer the optimal visible image for the thermal image.

That is, the neural network inputs the thermal image to the input layer and adjusts the first weight W1 and the second weight W2 so that the result output from the output layer approaches the visible image as the correct answer. Learn the trained model.

FIG. 4 is a schematic view showing an example of the structure of a trained model of an image conversion process for converting a thermal image into a visible image.
In the trained model shown in FIG. 4, the layer of the decoder portion and the layer of the encoder portion have a symmetrical structure, and have a U-Net structure connected by a skip connection.

Returning to FIG. 2, the learning-side learned model storage unit 114 stores the learning-side learned model, which is a learned model given by the model generation unit 113.

The learning side communication unit 115 sends the learning side learned model stored in the learning side learned model storage unit 114 to the posture estimation device 130.

The learning device 110 described above can be realized by a computer 160 as shown in FIG.
FIG. 5 is a block diagram schematically showing the configuration of the computer 160.
The computer 160 includes a communication device 161, an auxiliary storage device 162, a memory 163, and a processor 164.

The communication device 161 communicates data via a network, for example.
The auxiliary storage device 162 stores data and programs required for processing on the computer 160.
Memory 163 temporarily stores programs and data and provides a work area for processor 164.
The processor 164 reads the program stored in the auxiliary storage device 162 into the memory 163, and executes the program to execute the processing on the computer 160.

The learning side input unit 111 and the learning side communication unit 115 described above can be realized by the communication device 161.
The learned model storage unit 114 on the learning side can be realized by the auxiliary storage device 162.

The learning side data acquisition unit 112 and the model generation unit 113 can be realized by the processor 164 executing the program read into the memory 163. Such a program may be provided through a network, or may be recorded and provided on a recording medium. That is, such a program may be provided as, for example, a program product.

FIG. 6 is a flowchart showing a process of learning by the learning device 110.
First, the learning side data acquisition unit 112 acquires learning data via the learning side input unit 111 (S10). Here, it is assumed that the thermal image data, which is the image data of the thermal image, and the visible image data, which is the image data of the visible image, which are used as the training data, are acquired at the same time, and the first embodiment is such an example. Not limited to. If the thermal image data can be associated with the visible image data used as the correct answer for the thermal image data, they may be acquired at different timings. The acquired learning data is given to the model generation unit 113.

Next, the model generation unit 113 learns the visible image, which is the output corresponding to the thermal image, by so-called supervised learning based on the combination of the thermal image and the visible image shown in the training data, and obtains the trained model. Generate (S11).

Next, the learning side learned model storage unit 114 stores the generated learning model (S12). Then, the learning side communication unit 115 transmits the learning model to the posture estimation device 130.

FIG. 7 is a block diagram schematically showing the configuration of the posture estimation device 130.
The posture estimation device 130 includes an inference device 140 and a posture estimation execution device 150 that functions as a posture estimation unit.

The inference device 140 infers a visible image from a thermal image by using the learned model given by the learning device 110 as an inference side learning model.
The inference device 140 includes an inference side communication unit 141, an inference side learned model storage unit 142, an inference side input unit 143, an inference side data acquisition unit 144, and an inference unit 145.

The inference side communication unit 141 receives the learned model from the learning device 110, and stores the learned model as the inference side learned model in the inference side learned model storage unit 142.
The inference side trained model storage unit 142 is a storage unit that stores the inference side trained model.

The inference side input unit 143 is an input unit that accepts input of thermal image data indicating a thermal image of a subject. The thermal image data input here is also referred to as target thermal image data. Further, the thermal image represented by the target thermal image data is also referred to as a target thermal image, and the subject included in the target thermal image, which is the target for estimating the posture, is also referred to as the target subject.
The inference side data acquisition unit 144 is a data acquisition unit that acquires target thermal image data via the inference side input unit 143. The acquired target thermal image data is given to the inference unit 145.

The inference unit 145 infers a visible image of the target subject from the thermal image indicated by the target thermal image data by using the inference side trained model stored in the inference side trained model storage unit 142. In other words, the inference unit 145 inputs the thermal image indicated by the target thermal image data into the inference side trained model, and acquires the visible image corresponding to the thermal image inferred from the thermal image. Can be done. Then, the inference unit 145 generates visible image data indicating the inferred visible image, and gives the visible image data to the posture estimation execution device 150. The visible image data generated here is also referred to as target visible image data. Further, the visible image indicated by the target visible image data, in other words, the inferred visible image is also referred to as the target visible image.

The posture estimation execution device 150 estimates the posture of the subject existing in the visible image from the visible image indicated by the target visible image data. As a method of estimating the posture, a large amount of correspondence between the visible image and the posture of the person (for example, the positional relationship of parts) is learned in advance, and when the visible image is input, the person corresponding to the visible image There is a method of determining the posture based on the learning result.

The posture estimation device 130 described above can also be realized by a computer 160 as shown in FIG.
For example, the inference side communication unit 141 and the inference side input unit 143 can be realized by the communication device 161.
The inference side learned model storage unit 142 can be realized by the auxiliary storage device 162.

The inference side data acquisition unit 144 and the inference unit 145 can be realized by the processor 164 executing the program read into the memory 163. Such a program may be provided through a network, or may be recorded and provided on a recording medium. That is, such a program may be provided as, for example, a program product.

FIG. 8 is a flowchart showing a process in which the posture estimation device 130 infers a visible image corresponding to a thermal image and estimates the posture from the visible image.
First, the inference side data acquisition unit 144 acquires the target thermal image data indicating the thermal image via the inference side input unit 143 (S20). The acquired target thermal image data is given to the inference unit 145.

Next, the inference unit 145 inputs the thermal image indicated by the target thermal image data into the inference side trained model stored in the inference side trained model storage unit 142, and obtains a visible image corresponding to the thermal image. (S21).

Next, the inference unit 145 generates target visible image data indicating a visible image corresponding to the thermal image obtained by the inference side trained model, and gives the target visible image data to the posture estimation execution device 150 (S22). ).

Next, the posture estimation execution device 150 estimates the posture of the subject in the visible image indicated by the target visible image data (S23). Based on the posture estimated in this way, for example, it is possible to detect the abnormal behavior of the subject reflected in the thermal image.

As described above, according to the attitude estimation system 100 according to the first embodiment, the thermal image output from the thermal image sensor or the like is converted into a visible image, and the attitude estimation execution which is the learned attitude estimator for the visible image is executed. The orientation of the subject in the thermal image can be estimated using the device 150. Therefore, it is possible to estimate the posture using the existing learned posture estimator for visible images.

Further, when a posture estimator for a thermal image is used, it is necessary to learn the relationship between the thermal image and the posture, and it is necessary to annotate the posture to the thermal image. Manual annotation work on the thermal image cannot be performed with sufficient accuracy due to the lack of resolution of the thermal image. In the first embodiment, it is not necessary to use the posture estimator for the thermal image, so that these problems can be avoided.

In the first embodiment, the case where supervised learning is applied to the learning algorithm used by the model generation unit 113 has been described, but the first embodiment is not limited to such an example. For example, as a learning algorithm, reinforcement learning, unsupervised learning, semi-supervised learning, or the like can be used in addition to supervised learning.

Further, the model generation unit 113 may learn the visible image corresponding to the thermal image according to the learning data created for the plurality of posture estimation devices including the posture estimation device 130. The model generation unit 113 may acquire learning data from a plurality of posture estimation devices used in the same area, or may collect learning data from a plurality of posture estimation devices that operate independently in different areas. The visible image corresponding to the thermal image may be learned by using the data.

Further, the model generation unit 113 can add or remove the posture estimation device for collecting learning data to the target on the way.
Further, the model generation unit 113 applies the trained model in which the visible image corresponding to the thermal image is learned for one posture estimation device to another posture estimation device, and applies the trained model to another posture estimation device for the thermal image. The corresponding visible image may be retrained to update the trained model.

Further, as the learning algorithm used in the model generation unit 113, deep learning, which learns the extraction of the feature amount itself, can also be used. In addition, the model generation unit 113 may execute machine learning according to other known methods such as genetic programming, functional logic programming, or a support vector machine.

The learning device 110 and the inference device 140 are used to learn the visible image corresponding to the thermal image of the posture estimation system 100, and are connected to the posture estimation execution device 150 via a network, for example. It may be.
Further, the learning device 110, the inference device 140, or the posture estimation execution device 150 may exist on the cloud server.

Further, in the posture estimation system 100 according to the first embodiment described above, the learning device 110 and the posture estimation device 130 are separate devices. For example, the learning device 110 is provided in the posture estimation device 130. You may be. In such a case, the learning side communication unit 115 and the inference side communication unit 141 become unnecessary, and the learning side learned model storage unit 114 and the inference side learned model storage unit 142 can be integrated as the learned model storage unit. it can.

In the posture estimation system 100 according to the first embodiment, the posture estimation device 130 infers a visible image corresponding to a thermal image by using the trained model generated by the learning device 110. Is not limited to such an example. For example, the attitude estimation device 130 may acquire a trained model from the outside such as another system and infer a visible image corresponding to a thermal image based on the trained model.

Embodiment 2.
As shown in FIG. 1, the posture estimation system 200 according to the second embodiment includes a learning device 210 and a posture estimation device 230.

As shown in FIG. 2, the learning device 210 according to the second embodiment includes a learning side input unit 111, a learning side data acquisition unit 212, a model generation unit 213, and a learning side learned model storage unit 114. , The learning side communication unit 115 is provided.
The learning side input unit 111, the learning side learned model storage unit 114, and the learning side communication unit 115 of the learning device 210 in the second embodiment are the learning side input unit 111 and the learning side learned side of the learning device 110 in the first embodiment. This is the same as the model storage unit 114 and the learning side communication unit 115.

The learning side data acquisition unit 212 acquires learning data via the learning side input unit 111. The learning data acquired in the second embodiment is thermal image data indicating a thermal image, visible image data indicating a visible image which is a correct answer corresponding to the thermal image, and a correct answer corresponding to the visible image. Includes posture information indicating the posture of the subject. The acquired learning data is given to the model generation unit 213.

The model generation unit 213 learns the visible image corresponding to the thermal image and the posture corresponding to the visible image based on the learning data given from the learning side data acquisition unit 212. In other words, the model generation unit 213 learns the combination of the thermal image and the visible image shown in the training data and the combination of the visible image and the posture to infer the optimum posture corresponding to the thermal image. Generate a trained model. Specifically, the model generation unit 113 learns to infer the posture from the thermal image by learning the inference from the thermal image to the visible image and the inference from the visible image to the posture using the training data. Generate a finished model.
Then, the model generation unit 213 stores the generated learned model as the learning side trained model in the learning side trained model storage unit 114.

FIG. 9 is a block diagram schematically showing the configuration of the posture estimation device 230 according to the second embodiment.
The attitude estimation device 230 includes an inference side communication unit 141, an inference side learned model storage unit 142, an inference side input unit 143, an inference side data acquisition unit 144, and an inference unit 245.

The inference side communication unit 141, the inference side learned model storage unit 142, the inference side input unit 143, and the inference side data acquisition unit 144 of the attitude estimation device 230 according to the second embodiment are used to infer the attitude estimation device 130 according to the first embodiment. This is the same as the side communication unit 141, the inference side learned model storage unit 142, the inference side input unit 143, and the inference side data acquisition unit 144.

The inference unit 245 infers a visible image from the thermal image indicated by the target thermal image data by using the inference side trained model stored in the inference side trained model storage unit 142, and determines the posture from the visible image. Infer. In other words, the inference unit 145 estimates the posture of the subject existing in the thermal image, which is inferred from the thermal image, by inputting the thermal image indicated by the target thermal image data into the inference side trained model. To do.

The posture estimation device 230 described above can also be realized by a computer 160 as shown in FIG.
For example, the inference side communication unit 141 and the inference side input unit 143 can be realized by the communication device 161.
The inference side learned model storage unit 142 can be realized by the auxiliary storage device 162.

The inference side data acquisition unit 144 and the inference unit 245 can be realized by the processor 164 executing the program read into the memory 163. Such a program may be provided through a network, or may be recorded and provided on a recording medium. That is, such a program may be provided as, for example, a program product.

As described above, according to the posture estimation system 200 according to the second embodiment, it is possible to estimate the posture of the subject from the thermal image output from the thermal image sensor or the like. By inputting the visible image and the posture as teacher data at the time of learning, it is possible to avoid the work of annotating the posture information on the thermal image.

Further, unlike the first embodiment, the scale of the network can be suppressed and the amount of calculation can be reduced by not generating and outputting the visible image in the utilization phase.

Embodiment 3.
As shown in FIG. 1, the posture estimation system 300 according to the third embodiment includes a learning device 310 and a posture estimation device 330.

FIG. 10 is a block diagram schematically showing the configuration of the learning device 310.
The learning device 310 includes a learning side input unit 111, a learning side data acquisition unit 312, a model generation unit 313, a learning side learned model storage unit 114, a learning side communication unit 115, and a learning side contour extraction unit 316. To be equipped.

The learning side input unit 111, the learning side learned model storage unit 114, and the learning side communication unit 115 of the learning device 310 in the third embodiment are the learning side input unit 111 and the learning side learned side of the learning device 110 in the first embodiment. This is the same as the model storage unit 114 and the learning side communication unit 115.

The learning side data acquisition unit 312 acquires learning data via the learning side input unit 111. The acquired learning data is given to the model generation unit 313.
Further, the learning side data acquisition unit 312 gives the learning side contour extraction unit 316 the thermal image data indicating the thermal image included in the acquired learning data as the learning side thermal image data.

The learning side contour extraction unit 316 is a contour extraction unit that extracts a contour image showing the contour of the subject from the thermal image indicated by the learning side thermal image data. As the extraction method, there are a method using edge detection processing such as the Canny method or the Sobel method, or a method of combining binarization processing and edge detection. In the edge detection process, the edge of the subject is detected. Further, in the combination of the binarization process and the edge detection, the edge detection process may be performed after the binarization process is performed on the thermal image. Then, the learning side contour extraction unit 316 gives the contour image data indicating the extracted contour image to the model generation unit 313 as the learning side contour image data.

The model generation unit 313 learns a visible image corresponding to a thermal image based on the learning data given by the learning side data acquisition unit 312 and the learning side contour image data given by the learning side contour extraction unit 316. In other words, the model generation unit 313 learns the combination of the thermal image shown by the training data, the contour image shown by the learning side contour image data, and the visible image shown by the training data, thereby forming the thermal image and the thermal image. A trained model for inferring the optimum visible image corresponding to the contour image is generated. Specifically, the model generation unit 313 generates a trained model for inferring a visible image from a combination of thermal images and contour images by learning inference from a combination of thermal images and contour images to a visible image. To do.
Then, the model generation unit 313 stores the generated learned model as the learning side learned model in the learning side trained model storage unit 114.

FIG. 11 is a schematic view showing an example of the structure of the trained model of the image conversion process for converting the thermal image and the contour image into the visible image in the third embodiment.
The trained model shown in FIG. 11 has a U-Net structure in which the layer of the decoder portion and the layer of the encoder portion have a symmetrical structure and are connected by a skip connection. The decoder portion has two parallel paths, one for decoding the thermal image and the other for decoding the contour image.

As a result, in the trained model shown in FIG. 11, the decoder portion has two paths in parallel, one of which decodes the thermal image and the other of which decodes the contour image. The two vector information decoded in the center layer of the model is concatenated, and the concatenated information is input to the encoder part.
By having such a structure, in the third embodiment, the visible image converted from the thermal image contains a larger amount of edge components, and the accuracy of posture estimation can be improved.

The learning device 310 described above can also be realized by a computer 160 as shown in FIG.
For example, the learning side data acquisition unit 312, the model generation unit 313, and the learning side contour extraction unit 316 can also be realized by the processor 164 executing the program read into the memory 163. Such a program may be provided through a network, or may be recorded and provided on a recording medium. That is, such a program may be provided as, for example, a program product.

FIG. 12 is a block diagram schematically showing the configuration of the posture estimation device 330.
The posture estimation device 330 includes an inference device 340 and a posture estimation execution device 150.
The posture estimation execution device 150 of the posture estimation device 330 in the third embodiment is the same as the posture estimation execution device 150 in the first embodiment.

The inference device 340 includes an inference side communication unit 141, an inference side learned model storage unit 142, an inference side input unit 143, an inference side data acquisition unit 344, an inference side 345, and an inference side contour extraction unit 346. Be prepared.
The inference side communication unit 141, the inference side learned model storage unit 142 and the inference side input unit 143 of the inference device 340 in the third embodiment are the inference side communication unit 141 and the inference side trained in the inference device 140 in the first embodiment. This is the same as the model storage unit 142 and the inference side input unit 143.

The inference side data acquisition unit 144 acquires the target thermal image data via the inference side input unit 143. Then, the inference side data acquisition unit 144 gives the acquired target thermal image data to the inference unit 345 and the inference side contour extraction unit 346.

The inference side contour extraction unit 346 is a contour extraction unit that extracts a contour image from the thermal image indicated by the target thermal image data. The extraction method is the same as that of the learning side contour extraction unit 316. Then, the inference side contour extraction unit 346 gives the inference side contour image data to the inference unit 345 as contour image data indicating the extracted contour image. The contour image extracted here is also referred to as a target contour image, and the inference side contour image data is also referred to as a target contour image data.

The inference unit 345 uses the inference side trained model stored in the inference side trained model storage unit 142 from the combination of the thermal image shown in the target thermal image data and the contour image shown in the inference side contour image data. , Infer the visible image. In other words, the inference unit 345 infers from the thermal image by inputting the thermal image shown by the target thermal image data and the contour image indicated by the inference side contour image data into the inference side trained model. A visible image corresponding to a thermal image can be acquired. Then, the inference unit 345 generates visible image data indicating the inferred visible image, and gives the visible image data to the posture estimation execution device 150. The visible image data generated here is also referred to as target visible image data. The visible image indicated by the target visible image data is also referred to as a target visible image.

The posture estimation device 330 described above can also be realized by a computer 160 as shown in FIG.
For example, the inference side data acquisition unit 344, the inference side 345, and the inference side contour extraction unit 346 can be realized by the processor 164 executing the program read into the memory 163. Such a program may be provided through a network, or may be recorded and provided on a recording medium. That is, such a program may be provided as, for example, a program product.

In general, since the thermal image has ambiguous contour information, the visible image generated by using the trained model also has an ambiguous contour. Since contour information is important for posture estimation, the accuracy of posture estimation is reduced for images with ambiguous contours.
On the other hand, according to the posture estimation system 300 according to the third embodiment, by simultaneously inputting the thermal image and the contour image into the trained model, it is possible to generate a visible image in which the contour is not ambiguous. As a result, the attitude estimation accuracy from the generated visible image can be improved as compared with inputting the thermal image alone into the trained model.

100,200,300 posture estimation system, 110,210,310 learning device, 111 learning side input unit, 112,212,312 learning side data acquisition unit, 113,213,313 model generation unit, 114 learning side learned model storage Unit, 115 Learning side communication unit, 316 Learning side contour extraction unit, 130, 230, 330 Attitude estimation device, 140, 340 Inference device, 141 Inference side communication unit, 142 Inference side trained model storage unit, 143 Inference side input unit , 144,344 Inference side data acquisition unit, 145,245,345 Inference unit, 346 Inference side contour extraction unit, 150 Attitude estimation execution device.

Claims (14)

Includes a thermal image that images the temperature distribution of the subject by using infrared rays radiated from the subject, and a visible image that images the subject by using visible light reflected from the subject. A data acquisition unit that acquires training data,
A contour extraction unit that extracts a contour image showing the contour of the subject from the thermal image,
A model generation unit that generates a trained model for inferring the visible image from the combination of the thermal image and the contour image by learning inference from the combination of the thermal image and the contour image to the visible image. With ,
The trained model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel.
The learning device, characterized in that the two paths are a path for decoding the thermal image and a path for decoding the contour image . The learned model, a layer of the decoder portion, and layers of the encoder portion becomes symmetrical structure, according to claim 1, characterized in that it has a connected U-Net structure skip connection Learning device. In the central layer of the trained model, the vector in which the thermal image is decoded and the vector information in which the contour image is decoded are concatenated, and the concatenated information is input to the encoder portion.
2. The learning device according to claim 2. The learning device according to any one of claims 1 to 3, wherein the contour extraction unit extracts the contour image from the thermal image by an edge detection process for detecting the edge of the subject. Claims 1 to 3, wherein the contour extraction unit extracts the contour image from the thermal image by performing a binarization process on the thermal image and then performing an edge detection process . The learning device according to any one item . Includes a thermal image that images the temperature distribution of the subject by using infrared rays emitted from the subject, and a visible image that images the subject by using the visible light reflected from the subject. Using the training data and the contour image data showing the contour image showing the contour of the subject extracted from the thermal image, the inference from the combination of the thermal image and the contour image to the visible image is learned. A storage unit that stores a trained model for inferring the visible image from the combination of the thermal image and the contour image generated by the above.
A data acquisition unit that acquires target thermal image data indicating a target thermal image that is a thermal image of the target subject that is the target subject, and a data acquisition unit.
A contour extraction unit that extracts a target contour image, which is a contour image showing the contour of the target subject, from the target thermal image.
An inference unit that infers a target visible image, which is a visible image of the target subject, from a combination of the target thermal image and the target contour image using the trained model.
A posture estimation unit that estimates the posture of the target subject from the target visible image is provided .
The trained model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel.
The utilization device, characterized in that the two paths are a path for decoding the thermal image and a path for decoding the contour image . The learned model, a layer of the decoder portion, and layers of the encoder portion becomes symmetrical structure, according to claim 6, characterized in that it has a connected U-Net structure skip connection Utilization device. In the central layer of the trained model, the vector in which the thermal image is decoded and the vector information in which the contour image is decoded are concatenated, and the concatenated information is input to the encoder portion.
7. The utilization device according to claim 7. The utilization device according to any one of claims 6 to 8, wherein the contour extraction unit extracts the contour image from the thermal image by an edge detection process. Claims 6 to 8, wherein the contour extraction unit extracts the contour image from the thermal image by performing a binarization process on the thermal image and then performing an edge detection process . Utilization device according to any one item . Computer,
Includes a thermal image that images the temperature distribution of the subject by using infrared rays radiated from the subject, and a visible image that images the subject by using visible light reflected from the subject. Data acquisition unit that acquires training data,
A contour extraction unit that extracts a contour image showing the contour of the subject from the thermal image, and a contour extraction unit.
A model generation unit that generates a trained model for inferring the visible image from the combination of the thermal image and the contour image by learning the inference from the combination of the thermal image and the contour image to the visible image. to function as,
The trained model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel.
The two paths are a program for decoding the thermal image and a path for decoding the contour image . Computer,
It includes a thermal image that images the temperature distribution of the subject by using infrared rays emitted from the subject and a visible image that images the subject by using the visible light reflected from the subject. Using the training data and the contour image data showing the contour image showing the contour of the subject extracted from the thermal image, the inference from the combination of the thermal image and the contour image to the visible image is learned. A storage unit that stores a trained model for inferring the visible image from the combination of the thermal image and the contour image generated by the above.
A data acquisition unit that acquires target thermal image data indicating a target thermal image that is a thermal image of the target subject that is the target subject.
A contour extraction unit that extracts a target contour image, which is a contour image showing the contour of the target subject, from the target thermal image.
An inference unit that infers a target visible image, which is a visible image of the target subject, from a combination of the target thermal image and the target contour image using the trained model, and
It functions as a posture estimation unit that estimates the posture of the target subject from the target visible image .
The trained model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel.
The two paths are a program for decoding the thermal image and a path for decoding the contour image . Includes a thermal image that images the temperature distribution of the subject by using infrared rays radiated from the subject, and a visible image that images the subject by using visible light reflected from the subject. Get training data,
A contour image showing the contour of the subject is extracted from the thermal image,
It is a learning method for generating a trained model for inferring the visible image from the combination of the thermal image and the contour image by learning the inference from the combination of the thermal image and the contour image to the visible image. hand,
The trained model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel.
The learning method , wherein the two paths are a path for decoding the thermal image and a path for decoding the contour image . Acquires the target thermal image data indicating the target thermal image which is the thermal image of the target subject which is the target subject.
A target contour image, which is a contour image showing the contour of the target subject, is extracted from the target thermal image.
It includes a thermal image that images the temperature distribution of the subject by using infrared rays emitted from the subject and a visible image that images the subject by using the visible light reflected from the subject. Using the training data and the contour image data showing the contour image showing the contour of the subject extracted from the thermal image, the inference from the combination of the thermal image and the contour image to the visible image is learned. Using the trained model for inferring the visible image from the combination of the thermal image and the contour image generated by the above, the visible image of the target subject is obtained from the combination of the target thermal image and the target contour image. Infer the target visible image that is
It is a utilization method of estimating the posture of the target subject from the target visible image.
The trained model is formed by a layer of a decoder part and a layer of an encoder part, and the decoder part has two paths in parallel.
The utilization method, characterized in that the two paths are a path for decoding the thermal image and a path for decoding the contour image .

Images