JP2022081282A - Information processing device, information processing program, and information processing system - Google Patents

Abstract

To provide an information processing device capable of improving estimation accuracy in comfort of a space, an information processing program, and an information processing system.SOLUTION: In an information processing system in which a plurality of air conditioners, a plurality of sensor devices, a plurality of terminal devices, and an information processing device are communicably connected by wire or radio via a prescribed network N, an information processing device 100 comprises an estimation unit 132 which estimates comfort of a space in which shooting by a photographing device is impossible for a person in the space on the basis of information on a person region in which a person was imaged among the images shot in a space in which shooting is possible by the photographing device and which was identified on the basis of a posture of the person.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing apparatus, an information processing program and an information processing system.

Conventionally, techniques for providing a comfortable space for users have been known. For example, the temperature of the room air-conditioned by the air-conditioning system in the entire building is controlled to the air-conditioning target temperature. In addition, the room temperature of the room where the individual air conditioner is installed is detected by the temperature sensor, and the room temperature of the room where the individual air conditioner is installed is adjusted to the same room temperature as the room where the individual air conditioner is installed according to the detected room temperature. The technique of controlling to the target temperature is known.

Japanese Unexamined Patent Publication No. 2013-64537

There is a demand for a technique for improving the estimation accuracy of space comfort.

The information processing apparatus according to the embodiment is information about a person area, which is an area in which a person is captured in an image taken in a space that can be photographed by the photographing device, and is a person specified based on the posture of the person. It includes an estimation unit that estimates the comfort of the space for the person in a space that cannot be photographed by the photographing apparatus based on the information about the area.

FIG. 1 is a diagram showing a configuration example of an information processing system according to the first embodiment. FIG. 2 is a diagram for explaining an outline of information processing according to the first embodiment. FIG. 3 is a diagram showing a configuration example of the information processing apparatus according to the first embodiment. FIG. 4 is a diagram showing an example of a user information storage unit according to the first embodiment. FIG. 5 is a diagram showing an example of the environmental information storage unit according to the first embodiment. FIG. 6 is a diagram showing a configuration example of the information processing apparatus according to the second embodiment. FIG. 7 is a diagram for explaining an outline of information processing according to the second embodiment. FIG. 8 is a diagram for explaining an outline of information processing according to the second embodiment. FIG. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device.

Hereinafter, an embodiment for implementing the information processing apparatus, the information processing program, and the information processing system according to the present application (hereinafter referred to as “the embodiment”) will be described in detail with reference to the drawings. The information processing apparatus, information processing program, and information processing system according to the present application are not limited to this embodiment. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate explanations are omitted.

[1. First Embodiment]
[1-1. Introduction]
Conventionally, a technique for controlling air conditioning based on comfort in a thermal environment relating to the heat and cold of a person has been known. For example, PMV (Predicted Mean Vote), which consists of six factors that affect the heat balance of the human body, is known as an index for evaluating the comfort of a space. PMV consists of four environmental elements: temperature [° C], humidity [%], (heat) radiation [° C], and airflow [m / s], as well as the amount of clothing [clo] and the amount of metabolism (activity) [met]. It is calculated from two human factors. Here, the four environmental elements of temperature, humidity, radiation, and airflow can be obtained by sensing the space. On the other hand, as the two human factors of the amount of clothing and the amount of metabolism (activity amount), an estimated value estimated from the outside air temperature or the like was generally used. Therefore, the PMV value calculated based on the conventional method does not always reflect the comfort of the space actually felt by a person. That is, in the past, it was not always possible to accurately estimate the comfort of a space.

Therefore, the inventors according to the present invention use a machine learning model to obtain information on the environment such as the temperature and humidity of the space detected by an image taken by an imaging device such as an RGB camera or an infrared camera and an environment sensor. We are proposing a technology (hereinafter, also referred to as a proposed technology) that estimates the comfort of a space from (hereinafter, also referred to as environmental information). Specifically, the proposed technique estimates the comfort of a space based on information about a region in which a person is captured (hereinafter, also referred to as a person region) included in an image of the space. For example, the proposed technique estimates the comfort of a space based on information about a person area identified based on a posture estimation process that estimates the posture of an individual person. In this way, the proposed technique can precisely identify the human area from the image by using the posture estimation process, so that information on the amount of clothing and the amount of metabolism (activity) of the person to be estimated can be obtained. It can be estimated accurately. Further, since the proposed technique can accurately estimate two human factors such as the amount of clothing and the amount of metabolism (activity amount), it is possible to improve the estimation accuracy of the comfort of the space.

Here, the posture estimation process may be a process of estimating the posture of a person from a moving image or a still image and specifying a person area. The posture estimation process may be realized by any known technique. For example, the posture estimation process may be realized by image processing using a posture estimation technique (hereinafter, also referred to as a posture estimation technique). For example, the posture estimation process may be realized by a posture estimation technique that estimates the posture (skeleton) of a person or an animal from a moving image or a still image by using a deep learning model called a posture estimation model. When a plurality of people are shown in one image, the posture estimation process may detect feature points for the plurality of people to estimate the postures of the plurality of people. That is, the information processing apparatus 100 may estimate the comfort of the space in which each person is located for each of the plurality of persons.

For example, as an example of a posture estimation model, there is a method in which a feature point (also referred to as a key point) indicating a feature of a person's body included in an image is estimated and the posture of the person connecting the feature points is detected. For example, as an example of a posture estimation model, as a feature point of a person's body included in an image, a joint point indicating the position of a joint of the person's body is estimated, and the skeleton of the person's body generated by connecting the joint points. There is a skeleton model that detects the posture of a person. As described above, by using the posture estimation technique, it is possible to estimate the joint points and the skeleton model of the body of the person in the image, so that the part of the body of the person in the image can be estimated precisely. For example, among the types of posture estimation models that detect feature points, there is one that can estimate 30 types of parts of a person's body in an image. Specifically, using a posture estimation model that detects feature points, the head, eyes (right, left), ears (right, left), nose, neck, and shoulders are the body parts of the person in the image. (Right, center, left), elbow (right, left), spine, wrist (right, left), hand (right, left), thumb (right, left), hand (right, left), waist (right, center) , Left), knee (right, left), ankle (right, left), foot (right, left) can be identified.

Further, as an example of the posture estimation model, DensePose (reference URL: http://openaccess.thecvf.com/content_cvpr_2018/html/Guler_DensePose_Dense_Human_CVPR_2018_paper.html) is known. DensePose is a technique for detecting a person area of a person in a two-dimensional image and generating a three-dimensional body surface model corresponding to the detected person area. More specifically, DensePose is a technique for estimating the UV coordinates of a three-dimensional surface of a person in an RGB image by inputting an RGB image. By using DensePose, the UV coordinates of the 3D body surface can be estimated from the person area shown in the 2D image, so each human body part (part of the person's body) in the person area shown in the 2D image is delicate. Can be estimated to. With DensePose, it is possible to estimate 24 different parts of a person's body in an image. Specifically, when DensePose is used, the head (left, right), neck, torso, arms (left, right / upper, front / front, back), as the body parts of the person in the image from the RGB image, Legs (left, right / thighs, calves / front, back), hands (left, right), feet (left, right) can be identified.

However, in general, it may be difficult to install an imaging device such as an RGB camera or an infrared camera in a place where privacy needs to be considered (toilet, dressing room, etc.). Therefore, with the above-mentioned proposed technology, it may be difficult to estimate the comfort of a privacy space because it is difficult to obtain an image of a place where privacy needs to be considered (hereinafter, also referred to as a privacy space). there were. Further, in the above-mentioned proposed technique, it is necessary to install an imaging device such as an RGB camera and an infrared camera and an environment sensor in the same place. Therefore, the above-mentioned proposed technique has a problem that it requires a high equipment cost to install expensive equipment such as an RGB camera and an infrared camera in each space for which comfort is estimated, and the cost increases. rice field.

On the other hand, the information processing apparatus 100 according to the embodiment cannot be photographed by the photographing device based on the information regarding the person area, which is the area in which the person is captured, among the images captured in the space that can be photographed by the photographing device. Estimate the comfort of space for a person in a different space. As described above, in order to estimate the comfort of the space in a place where the photographing device is not installed, for example, the information processing apparatus 100 is photographed in a place where the photographing device is installed near the place where the photographing device is not installed. Use image information about images. For example, since a person in the facility can move freely, the information processing apparatus 100 uses the image information of the photographing device installed at the entrance / exit of the facility as information for estimating the comfort in the facility thereafter. Use as.

As a result, the information processing device 100 can accurately estimate the comfort of the space in a place where the photographing device is not installed. Further, the information processing device 100 can accurately estimate the comfort of the space in a place (toilet, dressing room, etc.) where the photographing device cannot be installed in order to protect the privacy of the user. That is, the information processing device 100 can improve the estimation accuracy of the comfort of the space in a place where the image pickup device cannot take an image. Further, the information processing apparatus 100 can reduce the number of imaging devices such as an RGB camera and an infrared camera installed. Therefore, the information processing apparatus 100 can reduce the cost for accurately estimating the comfort of the space.

Here, the space that can be photographed by the photographing device may be, for example, a space that can be photographed by the photographing device among the spaces in which the photographing device is installed. On the other hand, the space that cannot be photographed by the photographing device is, for example, a space that cannot be photographed by the photographing device because it is a blind spot of the photographing device, or a space that cannot be photographed by the photographing device because the photographing device is not installed. May be. That is, the space that cannot be photographed due to the photographing measures may be a space that cannot necessarily be photographed (captured if there is a photographing device). For example, as an example of a space that cannot be photographed by a photographing device, there is a space (toilet, dressing room, changing room, etc.) in which a photographing device is not installed in order to protect the privacy of the user. In the present disclosure, the photographing device is not particularly limited as long as it can acquire an image. For example, the photographing device may be a camera that acquires a still image, or may be a video camera that acquires a moving image and acquires a part of a frame as a still image. Further, for example, the photographing device may be a photographing module having a function capable of acquiring both a still image and a moving image, and acquiring a still image while appropriately switching these functions.

Further, the person area may be any area as long as the information processing apparatus 100 can identify the person to be estimated. For example, the person area may include an area indicating the space around the person in addition to the area indicating the person. For example, the person area may include an area in which the space around the person is imaged in addition to the area in which the person is imaged. For example, the person area may include an area indicating furniture or the like existing around the person. In the following, a person located in a space may be referred to as a "user".

[1-2. Information processing system configuration]
First, the configuration of the information processing system 1 according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram showing a configuration example of the information processing system 1 according to the first embodiment. The information processing system 1 may include an air conditioner 10, an RGB camera 20, an infrared camera 30, an environment sensor 40, an authentication device 50, and an information processing device 100. The air conditioner 10, the RGB camera 20, the infrared camera 30, the environment sensor 40, the authentication device 50, and the information processing device 100 may be communicably connected via a predetermined network N via a predetermined network N. .. The information processing system 1 includes an arbitrary number of air conditioning devices 10, an arbitrary number of RGB cameras 20, an arbitrary number of infrared cameras 30, an arbitrary number of environment sensors 40, and an arbitrary number of authentications. The device 50 and an arbitrary number of information processing devices 100 may be included. The RGB camera 20 and the infrared camera 30 may be an example of a photographing device.

The air conditioner 10 is a device that adjusts the temperature, humidity, and the like of the air in the space. For example, the air conditioner 10 may be an air conditioner installed in each space subject to air conditioning control by the information processing device 100. For example, the air conditioner 10 may be an air conditioner installed for each place (for example, a room) in a space subject to air conditioning control by the information processing device 100. Further, the air conditioner 10 may be able to communicate with other devices by the IoT (Internet of Things) technology. The air conditioner 10 can adjust the air conditioning of the space to be controlled according to the control of the information processing device 100. Specifically, first, the air conditioner 10 may receive control information regarding air conditioning control from the information processing device 100. For example, the air conditioner 10 may receive information on a set temperature and a set humidity set for air conditioning from the information processing device 100. Subsequently, the air conditioner 10 may adjust the air conditioning of the space according to the control information received from the information processing device 100. For example, when the air conditioner 10 receives information on the set temperature and the set humidity, the air conditioner 10 may adjust the air conditioning so that the temperature and humidity of the space are maintained at the set temperature and the set humidity.

The RGB camera 20 is a device that acquires an RGB image of space. Specifically, the RGB camera 20 may acquire an RGB image of a space that can be photographed by the RGB camera 20. For example, the RGB camera 20 may acquire an RGB image each time a new person is detected in a space where the image can be taken. For example, the RGB camera 20 may always acquire an RGB image. Further, the RGB camera 20 may be installed in a space subject to air conditioning control by the information processing device 100. For example, the RGB camera 20 may be installed at the entrance or exit of a facility (for example, a building or a commercial facility in which a large number of offices are located) subject to air conditioning control by the information processing device 100. Further, the RGB camera 20 may capture the same space as the infrared camera 30. Further, the RGB camera 20 may be able to communicate with other devices by the IoT technique. For example, when the RGB camera 20 acquires an RGB image, the acquired RGB image may be transmitted to the information processing apparatus 100.

The infrared camera 30 is a device that acquires a thermal image of space. Specifically, the infrared camera 30 may acquire a thermal image of a space that can be photographed by the infrared camera 30. For example, the infrared camera 30 may acquire a thermal image every time a new person is detected in a space where the image can be taken. For example, the infrared camera 30 may constantly acquire a thermal image. Further, the infrared camera 30 may be installed in a space subject to air conditioning control by the information processing device 100. For example, the infrared camera 30 may be installed at an entrance or an exit of a facility subject to air conditioning control by the information processing device 100. Further, the infrared camera 30 may capture the same space as the RGB camera 20. Further, the infrared camera 30 may be able to communicate with other devices by the IoT technique. For example, when the infrared camera 30 acquires a thermal image, the acquired thermal image may be transmitted to the information processing apparatus 100.

The environment sensor 40 is a sensor device that acquires environmental information related to the environment such as the temperature and humidity of the space. Specifically, the environment sensor 40 may detect the temperature and humidity of the space in which the environment sensor 40 is installed. The environment sensor 40 may be a USB type environment sensor. Further, at least one environment sensor 40 may be installed in a space subject to air conditioning control by the information processing apparatus 100. For example, the environment sensor 40 may be installed every few meters in a facility subject to air conditioning control by the information processing device 100. Further, the environment sensor 40 may be able to communicate with other devices by the IoT technique. For example, when the environment sensor 40 detects the temperature and humidity of the space, the environment sensor 40 may transmit information on the detected temperature and humidity of the space to the information processing apparatus 100. The environmental information acquired by the environment sensor 40 may be any information that can be used in the determination of comfort described later, and is not limited to the above example. For example, the environment sensor 40 may acquire arbitrary information such as atmospheric pressure and illuminance as environmental information.

The authentication device 50 is a device for personally authenticating a user. The authentication device 50 may be installed at an entrance or an exit of a space subject to air conditioning control by the information processing device 100. The authentication device 50 may acquire the user's personal authentication information by using a sensor. Subsequently, when the authentication device 50 acquires the personal authentication information, the authentication device 50 collates the personal authentication information stored in advance with the acquired personal authentication information and performs personal authentication. Further, the authentication device 50 may be able to communicate with other devices by the IoT technique. For example, the authentication device 50 may acquire personal authentication information stored in advance in the information processing device 100 from the information processing device 100. Further, when the personal authentication is performed, the authentication device 50 may transmit the authentication result of the personal authentication to the information processing device 100. For example, when the personal authentication is successful, the authentication device 50 provides the information processing device 100 with information indicating that the personal authentication is successful and information that can identify a user who has succeeded in the personal authentication (for example, a user ID). You may send it.

The authentication device 50 according to the first embodiment is a face recognition device. The authentication device 50, which is a face authentication device, may acquire user's face image information by using an RGB camera as an example of personal authentication information. Subsequently, when the user's face image information is acquired, the authentication device 50 collates the face image information stored in advance with the acquired face image information to perform face authentication. For example, the face recognition device may acquire the user's face image information based on the image acquired from the RGB camera 20. Further, when the face recognition is successful, the authentication device 50 transmits the face image information of the user who can identify the user who has succeeded in the face recognition to the information processing device 100 together with the authentication information indicating that the face recognition is successful. You can do it.

The information processing device 100 is an information processing device that executes information processing according to the first embodiment. Specifically, the information processing apparatus 100 is a machine learning model learned to output the comfort of the space for the user as the output information when the information about the person area is input as the input information (hereinafter, comfort). An estimation model) may be used to estimate the comfort of a person in a space that cannot be photographed by an imaging device. For example, the information processing device 100 inputs information about a person area photographed in a space that can be photographed by the photographing device into a comfort estimation model, and obtains the comfort of the space for the person in a space that cannot be photographed by the photographing device. It may be output as output information of the comfort estimation model.

Further, the information processing device 100 is a control device that controls the air conditioning device 10. Specifically, the information processing device 100 may control the air conditioner 10 installed at the place where the user is located, based on the estimated comfort of the space. For example, when the estimated comfort of the space is "cold", the information processing apparatus 100 may control the air conditioner 10 so as to raise the temperature at the place. Further, when the estimated comfort of the space is "normal", the information processing device 100 may control the air conditioner 10 so as to maintain the temperature at the place. Further, when the estimated comfort of the space is "hot", the information processing device 100 may control the air conditioner 10 so as to lower the temperature at the place.

[1-3. Information processing overview]
Next, the outline of the information processing according to the first embodiment will be described with reference to FIG. FIG. 2 is a diagram for explaining an outline of information processing according to the first embodiment. Reference numerals P11 to P16 shown in FIG. 2 indicate six locations in the building subject to air conditioning control by the information processing apparatus 100, respectively. In the following, the place indicated by "P1 *" (* = 1 to 6) may be described as "place P1 *" (* = 1 to 6).

FIG. 2 illustrates a case where the user U1 sequentially moves from place P11 to place P16 in a building (an example of space) subject to air conditioning control by the information processing apparatus 100. For example, the user U1 moves from the place P11, which is the entrance near the entrance of the building, to the place P12, which is the corridor leading to the place P11. After that, the user U1 moves from the place P12 which is a corridor to the place P13 which is a shared space located in the back of the place P12. After that, the user U1 moves from the shared space P13 to the first conference room P14 adjacent to the place P13. After that, the user U1 moves from the place P14, which is the first conference room, to the place P15, which is the office room adjacent to the place P14. After that, the user U1 moves from the office P15 to the second conference room P16 adjacent to the office P15.

Further, the check mark shown in FIG. 2 indicates a place where the RGB camera 20, the infrared camera 30, or the environment sensor 40 is installed. For example, the place P11, the place P14, and the place P16 indicate that the place where the RGB camera 20, the infrared camera 30, and the environment sensor 40 are installed. On the other hand, the place P12, the place P13, and the place P15 indicate that the RGB camera 20 and the infrared camera 30 are not installed, but only the environment sensor 40 is installed. Although not shown, it is assumed that the authentication device 50, which is a face authentication device, is installed at the place P11. Further, in the following description, a photographing device such as an RGB camera 20 and an infrared camera 30 may be simply referred to as a “camera”.

Further, in the following, the RGB camera 20 may be described separately as the RGB camera 20- * (* = 1, 4, 6) depending on the space in which the RGB camera 20 is installed. For example, the RGB camera 20-1 is an RGB camera 20 installed at the location P11. Further, for example, the RGB camera 20-4 is an RGB camera 20 installed at the place P14. Further, in the following, when the RGB camera 20- * (* = 1, 4, 6) is described without particular distinction, it is described as the RGB camera 20.

Further, in the following, the infrared camera 30 may be described separately as the infrared camera 30- * (* = 1, 4, 6) depending on the space in which the infrared camera 30 is installed. For example, the infrared camera 30-1 is an infrared camera 30 installed at the place P11. Further, for example, the infrared camera 30-4 is an infrared camera 30 installed at the place P14. Further, in the following, when the infrared camera 30- * (* = 1, 4, 6) is described without particular distinction, it is described as the infrared camera 30.

Further, in the following, the environment sensor 40 may be described separately as the environment sensor 40- * (* = 1 to 6) depending on the space in which the environment sensor 40 is installed. For example, the environment sensor 40-1 is an environment sensor 40 installed at the place P11. Further, for example, the environment sensor 40-2 is an environment sensor 40 installed at the place P12. Further, in the following, when the environment sensor 40- * (* = 1 to 6) is described without particular distinction, it is described as the environment sensor 40.

Further, in the following, the authentication device 50 may be described separately as the authentication device 50- * (* = 1) depending on the space in which the authentication device 50 is installed. For example, the authentication device 50-1 is an authentication device 50 installed at the place P11. Further, in the following, when the authentication device 50- * (* = 1) is described without particular distinction, it is described as the authentication device 50.

First, in FIG. 2, the user U1 visits the place P11 where the camera is installed. At this time, the information processing apparatus 100 installs the user ID "U1" that identifies the user U1 and the face image information # 11 of the user U1 at the place P11 together with the authentication information that the face authentication of the user U1 is successful. It may be obtained from the authentication device 50-1.

Subsequently, when the information processing device 100 acquires the authentication information from the authentication device 50-1, the RGB image # 11 taken by the RGB camera 20-1 installed at the location P11 is acquired from the RGB camera 20-1. good. Subsequently, the information processing apparatus 100 may specify the person (area) newly reflected in the RGB image # 11 as the user U1 identified by the user ID “U1”.

Further, when the information processing apparatus 100 acquires the RGB image # 11, it may generate image information related to the RGB image # 11 (hereinafter, also referred to as RGB image information # 11). For example, when the information processing apparatus 100 acquires the RGB image # 11, the person area R11 of the user U1 in the RGB image # 11 may be specified by the posture estimation process. Subsequently, when the information processing apparatus 100 identifies the person region R11, the learning model M1 may be used as an example of the RGB image information # 11 to generate a feature vector V11 showing the characteristics of the person region R11.

Further, the information processing apparatus 100 may acquire the thermal image # 11 taken by the infrared camera 30-1 installed at the place P11 from the infrared camera 30-1. When the information processing apparatus 100 acquires the thermal image # 11, it may generate image information related to the thermal image # 11 (hereinafter, also referred to as thermal image information # 11). For example, when the information processing apparatus 100 acquires the thermal image # 11, the information processing apparatus 100 may specify the person area R12 of the user U1 in the thermal image # 11 based on the posture estimation process. Subsequently, when the information processing apparatus 100 identifies the person region R12, the learning model M2 may be used as an example of the thermal image information # 11 to generate a feature vector V12 showing the characteristics of the person region R12.

Further, when the information processing apparatus 100 generates RGB image information # 11 (feature vector V11) and thermal image information # 11 (feature vector V12), RGB image information # 11 (feature vector V11) and thermal image information # 11 (feature vector V11) are generated. The feature vector V12) and the face image information # 11 of the user U1 acquired from the authentication device 50-1 may be stored in association with each other.

Further, the information processing apparatus 100 acquires environmental information (hereinafter, also referred to as environmental information # 11) regarding the temperature and humidity at that time detected by the environment sensor 40-1 installed at the place P11 from the environment sensor 40-1. It's okay. When the information processing apparatus 100 acquires the environment information # 11, the information processing apparatus 100 may generate a feature vector V13 indicating the characteristics of the environment information # 11 by using the learning model M3.

Further, when the information processing apparatus 100 acquires the environment information # 11, the sensor ID "40-1" that identifies the environment sensor 40-1 is stored in association with the acquisition time of the environment information # 11 and the environment information # 11. You can do it.

Subsequently, the information processing apparatus 100 may estimate the comfort of the space for the user U1 at the place P11 where the camera is installed by using the comfort estimation model M4. For example, the information processing apparatus 100 inputs the feature vector V11, the feature vector V12, and the feature vector V13 into the comfort estimation model M4 to comfort the space comfort for the user U1 at the place P11 where the camera is installed. It may be output as output information of the estimation model M4.

Next, in FIG. 2, the user U1 may move from the place P11 where the camera is installed to the place P12 where the camera is not installed. At this time, the information processing apparatus 100 acquires environmental information (hereinafter, also referred to as environmental information # 12) regarding the temperature and humidity at that time detected by the environment sensor 40-2 installed at the place P12 from the environment sensor 40-2. You can do it. When the information processing apparatus 100 acquires the environment information # 12, the information processing apparatus 100 may generate a feature vector V23 indicating the characteristics of the environment information # 12 by using the learning model M3. Further, when the information processing apparatus 100 acquires the environment information # 12, the sensor ID “40-2” that identifies the environment sensor 40-2, the environment information # 12, and the acquisition time of the environment information # 12 are stored in association with each other. You can do it.

Subsequently, the information processing apparatus 100 may estimate the comfort of the space for the user U1 at the place P12 where the camera is not installed by using the comfort estimation model M4. Specifically, in the information processing apparatus 100, a camera is installed using a feature vector V11 showing the characteristics of the person area R11 in the place P11 where the camera is installed and a feature vector V12 showing the features of the person area R12. The comfort of the space for the user U1 in the absence P12 may be estimated. For example, the information processing apparatus 100 inputs the feature vector V11, the feature vector V12, and the feature vector V23 into the comfort estimation model M4 to comfort the space comfort for the user U1 in the place P12 where the camera is not installed. It may be output as output information of the estimation model M4.

As described above, in the information processing apparatus 100, the camera is installed based on the RGB image information and the thermal image information of the user U1 in the place P11 where the camera is installed near the place P12 where the camera is not installed. Estimate the comfort of the space for the user U1 in the absence P12. Here, since the place P12 where the camera is not installed and the place P11 where the camera is installed are close to each other, the information on the amount of clothing and the amount of metabolism (activity amount) of the user U1 at the place P11 and the place P12 is almost the same. Can be regarded as. Therefore, the information processing apparatus 100 can accurately estimate the comfort of the space for the user U1 in the place P12 where the camera is not installed.

Next, in FIG. 2, the user U1 moves from the place P12 where the camera is not installed to the place P13 where the camera is not installed. At this time, the information processing apparatus 100 acquires environmental information (hereinafter, also referred to as environmental information # 13) regarding the temperature and humidity at that time detected by the environment sensor 40-3 installed at the place P13 from the environment sensor 40-3. You can do it. When the information processing apparatus 100 acquires the environment information # 13, the information processing apparatus 100 may generate a feature vector V33 indicating the characteristics of the environment information # 13 by using the learning model M3. Further, when the information processing apparatus 100 acquires the environment information # 13, the sensor ID "40-3" that identifies the environment sensor 40-3, the environment information # 13, and the acquisition time of the environment information # 13 are stored in association with each other. You can do it.

Subsequently, the information processing apparatus 100 may estimate the comfort of the space for the user U1 at the place P13 where the camera is not installed by using the comfort estimation model M4. Specifically, in the information processing apparatus 100, a camera is installed using a feature vector V11 showing the characteristics of the person area R11 in the place P11 where the camera is installed and a feature vector V12 showing the features of the person area R12. The comfort of the space for the user U1 in the absence P13 may be estimated. For example, the information processing apparatus 100 inputs the feature vector V11, the feature vector V12, and the feature vector V33 into the comfort estimation model M4 to comfort the space comfort for the user U1 in the place P13 where the camera is not installed. It may be output as output information of the estimation model M4.

As described above, in the information processing apparatus 100, the camera is installed based on the RGB image information and the thermal image information of the user U1 in the place P11 where the camera is installed near the place P13 where the camera is not installed. Estimate the comfort of the space for the user U1 in the absence P13. Here, since the place P13 where the camera is not installed and the place P11 where the camera is installed are close to each other, the information on the amount of clothing and the amount of metabolism (activity amount) of the user U1 at the place P11 and the place P13 is almost the same. Can be regarded as. Therefore, the information processing apparatus 100 can accurately estimate the comfort of the space for the user U1 in the place P13 where the camera is not installed.

Next, in FIG. 2, the user U1 moves from the place P13 where the camera is not installed to the place P14 where the camera is installed. At this time, the information processing apparatus 100 may acquire the RGB image # 14 taken by the RGB camera 20-4 installed at the place P14 from the RGB camera 20-4. Subsequently, when the information processing apparatus 100 acquires the RGB image # 14, the face image information of the user U1 included in the RGB image # 14 and the face image information # 11 of the user U1 acquired from the authentication device 50-1 are obtained. May be collated to identify the user U1 included in the RGB image # 14.

Further, when the information processing apparatus 100 acquires the RGB image # 14, it may generate image information related to the RGB image # 14 (hereinafter, also referred to as RGB image information # 14). For example, when the information processing apparatus 100 acquires the RGB image # 14, the information processing apparatus 100 may specify the person area R14 of the user U1 in the RGB image # 14 by the posture estimation process. Subsequently, when the information processing apparatus 100 identifies the person region R14, the learning model M1 may be used as an example of the RGB image information # 14 to generate a feature vector V41 showing the characteristics of the person region R14.

Further, the information processing apparatus 100 may acquire the thermal image # 14 taken by the infrared camera 30-4 installed at the place P14 from the infrared camera 30-4. When the information processing apparatus 100 acquires the thermal image # 14, it may generate image information related to the thermal image # 14 (hereinafter, also referred to as thermal image information # 14). For example, when the information processing apparatus 100 acquires the thermal image # 14, the information processing apparatus 100 may specify the person area R15 of the user U1 in the thermal image # 14 based on the posture estimation process. Subsequently, when the information processing apparatus 100 identifies the person region R15, the learning model M2 may be used as an example of the thermal image information # 14 to generate a feature vector V42 showing the characteristics of the person region R15.

Further, when the information processing apparatus 100 generates the RGB image information # 14 (feature vector V41) and the thermal image information # 14 (feature vector V42), the RGB image information stored in association with the image information # 1 of the user U1 is stored. # 11 (feature vector V11) and thermal image information # 11 (feature vector V12) may be deleted. Subsequently, the information processing apparatus 100 obtains RGB image information # 14 (feature vector V41), thermal image information # 14 (feature vector V42), and user U1 face image information # 11 acquired from the authentication device 50-1. It may be associated and stored.

As described above, each time the information processing apparatus 100 acquires RGB image information or thermal image information, the information processing apparatus 100 may update the RGB image information or thermal image information stored in association with the user's face image information.

Further, the information processing apparatus 100 acquires environmental information (hereinafter, also referred to as environmental information # 14) regarding the temperature and humidity at that time detected by the environment sensor 40-4 installed at the place P14 from the environment sensor 40-4. It's okay. When the information processing apparatus 100 acquires the environment information # 14, the information processing apparatus 100 may generate a feature vector V43 indicating the characteristics of the environment information # 14 by using the learning model M3.

Further, when the information processing apparatus 100 acquires the environment information # 14, the sensor ID “40-4” that identifies the environment sensor 40-4, the environment information # 14, and the acquisition time of the environment information # 14 are stored in association with each other. You can do it.

Subsequently, the information processing apparatus 100 may estimate the comfort of the space for the user U1 at the place P14 where the camera is installed by using the comfort estimation model M4. For example, the information processing apparatus 100 inputs the feature vector V41, the feature vector V42, and the feature vector V43 into the comfort estimation model M4 to comfort the space comfort for the user U1 at the place P14 where the camera is installed. It may be output as output information of the estimation model M4.

Next, in FIG. 2, the user U1 moves from the place P14 where the camera is installed to the place P15 where the camera is not installed. At this time, the information processing apparatus 100 acquires environmental information (hereinafter, also referred to as environmental information # 15) regarding the temperature and humidity at that time detected by the environment sensor 40-5 installed at the place P15 from the environment sensor 40-5. You can do it. When the information processing apparatus 100 acquires the environment information # 15, the information processing apparatus 100 may generate a feature vector V53 indicating the characteristics of the environment information # 15 by using the learning model M3. Further, when the information processing apparatus 100 acquires the environment information # 15, the sensor ID "40-5" that identifies the environment sensor 40-5, the environment information # 15, and the acquisition time of the environment information # 15 are stored in association with each other. You can do it.

Subsequently, the information processing apparatus 100 may estimate the comfort of the space for the user U1 at the place P15 where the camera is not installed by using the comfort estimation model M4. Specifically, in the information processing apparatus 100, a camera is installed using a feature vector V41 showing the characteristics of the person area R14 and a feature vector V42 showing the features of the person area R15 in the place P14 where the camera is installed. The comfort of the space for the user U1 in the absence P15 may be estimated. For example, the information processing apparatus 100 inputs the feature vector V41, the feature vector V42, and the feature vector V53 into the comfort estimation model M4 to comfort the space comfort for the user U1 in the place P15 where the camera is not installed. It may be output as output information of the estimation model M4.

As described above, in the information processing apparatus 100, the camera is installed based on the RGB image information and the thermal image information of the user U1 in the place P14 where the camera is installed near the place P15 where the camera is not installed. Estimate the comfort of the space for the user U1 in the absence P15. Here, since the place P15 where the camera is not installed and the place P14 where the camera is installed are close to each other, the information on the amount of clothing and the amount of metabolism (activity amount) of the user U1 at the place P14 and the place P15 is almost the same. Can be regarded as. Therefore, the information processing apparatus 100 can accurately estimate the comfort of the space for the user U1 in the place P15 where the camera is not installed.

Further, the information processing apparatus 100 can estimate the comfort of the space for the user U1 based on the latest RGB image information and the thermal image information of the user U1. Therefore, the information processing apparatus 100 has the latest clothing amount and metabolic amount (activity amount) of the user U1 even when the information regarding the clothing amount and the metabolic amount (activity amount) of the user U1 changes with the passage of time. Based on the information about, the comfort of the space for the user U1 can be estimated. Therefore, the information processing apparatus 100 can accurately estimate the comfort of the space for the user U1 in the place P15 where the camera is not installed.

Next, in FIG. 2, the user U1 moves from the place P15 where the camera is not installed to the place P16 where the camera is installed. At this time, the information processing apparatus 100 may acquire the RGB image # 16 taken by the RGB cameras 20-6 installed at the place P16 from the RGB cameras 20-6. Subsequently, when the information processing apparatus 100 acquires the RGB image # 16, the face image information of the user U1 included in the RGB image # 16 and the face image information # 11 of the user U1 acquired from the authentication device 50-1 are obtained. May be collated to identify the user U1 included in the RGB image # 16.

Further, when the information processing apparatus 100 acquires the RGB image # 16, it may generate image information related to the RGB image # 16 (hereinafter, also referred to as RGB image information # 16). For example, when the information processing apparatus 100 acquires the RGB image # 16, the information processing apparatus 100 may specify the person area R16 of the user U1 in the RGB image # 16 by the posture estimation process. Subsequently, when the information processing apparatus 100 specifies the person region R16, the learning model M1 may be used as an example of the RGB image information # 16 to generate a feature vector V61 showing the characteristics of the person region R16.

Further, the information processing apparatus 100 may acquire the thermal image # 16 taken by the infrared camera 30-6 installed at the place P16 from the infrared camera 30-6. When the information processing apparatus 100 acquires the thermal image # 16, it may generate image information related to the thermal image # 16 (hereinafter, also referred to as thermal image information # 16). For example, when the information processing apparatus 100 acquires the thermal image # 16, the information processing apparatus 100 may specify the person area R17 of the user U1 in the thermal image # 16 based on the posture estimation process. Subsequently, when the information processing apparatus 100 identifies the person region R17, the learning model M2 may be used as an example of the thermal image information # 16 to generate a feature vector V62 showing the characteristics of the person region R17.

Further, when the information processing apparatus 100 generates the RGB image information # 16 (feature vector V61) and the thermal image information # 16 (feature vector V62), the RGB image information stored in association with the image information # 1 of the user U1 is stored. # 14 (feature vector V41) and thermal image information # 14 (feature vector V42) may be deleted. Subsequently, the information processing apparatus 100 obtains RGB image information # 16 (feature vector V61), thermal image information # 16 (feature vector V62), and user U1 face image information # 11 acquired from the authentication device 50-1. It may be associated and stored.

In this way, each time the information processing apparatus 100 acquires RGB image information or thermal image information, the information processing apparatus 100 updates the RGB image information or thermal image information stored in association with the user's face image information.

Further, the information processing apparatus 100 acquires environmental information (hereinafter, also referred to as environmental information # 16) regarding the temperature and humidity at that time detected by the environment sensor 40-6 installed at the place P16 from the environment sensor 40-6. It's okay. When the information processing apparatus 100 acquires the environment information # 16, the information processing apparatus 100 may generate a feature vector V63 indicating the features of the environment information # 16 by using the learning model M3.

Further, when the information processing apparatus 100 acquires the environment information # 16, the sensor ID "40-6" that identifies the environment sensor 40-6, the environment information # 16, and the acquisition time of the environment information # 16 are stored in association with each other. You can do it.

Subsequently, the information processing apparatus 100 may estimate the comfort of the space for the user U1 at the place P16 where the camera is installed by using the comfort estimation model M4. For example, the information processing apparatus 100 inputs the feature vector V61, the feature vector V62, and the feature vector V63 into the comfort estimation model M4 to make the space comfort for the user U1 at the place P16 where the camera is installed comfortable. It may be output as output information of the estimation model M4.

[1-4. Information processing device configuration]
Next, the configuration of the information processing apparatus 100 according to the first embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a configuration example of the information processing apparatus 100 according to the first embodiment. The information processing device 100 may include a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 includes an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from the administrator of the information processing device 100, and a display unit (for example, a liquid crystal display, etc.) for displaying various information. You may have.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 110 is connected to the network by wire or wirelessly, and transmits / receives information to / from, for example, the air conditioner 10, the RGB camera 20, the infrared camera 30, and the environment sensor 40.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. For example, the storage unit 120 may store information about various learning models. For example, the storage unit 120 may store information about the learning models M1 to M3 and the comfort estimation model M4 as various learning models. For example, the storage unit 120 may store information regarding the connection coefficients of various learning models.

(User information storage unit 121)
The user information storage unit 121 may store various types of information about the user. Here, an example of the user information storage unit 121 according to the first embodiment will be described with reference to FIG. FIG. 4 is a diagram showing an example of the user information storage unit 121 according to the first embodiment.

In the example shown in FIG. 4, the user information storage unit 121 has RGB image information about an RGB image taken in a space that can be photographed by the RGB camera 20, and a user's face acquired from an authentication device 50 that is a face recognition device. It may be stored in association with the image information. For example, the user information storage unit 121 stores, as an example of RGB image information, a feature vector V1 indicating the characteristics of the user's person area R1 included in the RGB image and the user's face image information in association with each other. good.

Further, the user information storage unit 121 corresponds between the thermal image information regarding the thermal image taken in the space that can be photographed by the infrared camera 30 and the user's face image information acquired from the authentication device 50 which is a face authentication device. You may attach it and memorize it. For example, the user information storage unit 121 stores, as an example of the thermal image information, the feature vector V2 showing the characteristics of the user's person area R2 included in the thermal image and the user's face image information in association with each other. good.

Further, the user information storage unit 121 may store the image information related to the image having the latest shooting time among the two or more images having different shooting times in association with the information that can identify the user.

Note that FIG. 4 describes, as an example of information that can identify the user, an example in which the face image information for user face authentication acquired from the authentication device 50, which is a face authentication device, and the image information are stored in association with each other. However, the information that can identify the user is not limited to the user's facial image information. For example, the user information storage unit 121 is an example of information that can identify a user, and is used for biometric authentication information (for fingerprint authentication information or iris authentication) used for a user's biometric authentication process such as fingerprint authentication or iris authentication. Information etc.) and image information may be associated and stored. Further, as an example of information that can identify the user, the user information storage unit 121 includes item authentication information (IC card authentication information, etc.) and image information used for the user's item authentication process such as IC card authentication. May be stored in association with.

(Environmental information storage unit 122)
The environmental information storage unit 122 may store the environmental information detected by the environmental sensor. Here, an example of the environmental information storage unit 122 according to the first embodiment will be described with reference to FIG. FIG. 5 is a diagram showing an example of the environmental information storage unit 122 according to the first embodiment. In the example shown in FIG. 5, the environment information storage unit 122 may store the sensor ID that can identify the environment sensor, the time when the environment information is detected, and the environment information in association with each other. Further, the environmental information storage unit 122 may store a log of environmental information detected by the environmental sensor at predetermined time intervals.

(Control unit 130)
The control unit 130 is a controller, and for example, various programs (information processing programs) stored in a storage device inside the information processing device 100 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. (Corresponding to one example) is realized by executing the RAM as a work area. Further, the control unit 130 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 130 has an acquisition unit 131, an estimation unit 132, and an air conditioning control unit 133 as functional units, and realizes or executes the operation of information processing described below. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 3, and may be any other configuration as long as it is configured to perform information processing described later. Further, each functional unit indicates the function of the control unit 130, and does not necessarily have to be physically distinguished.

(Acquisition unit 131)
The acquisition unit 131 may acquire the user ID that identifies the user and the face image information of the user used for face authentication. For example, the acquisition unit 131 may acquire the user ID and the user's face image information from the authentication device 50, which is a face authentication device. When the acquisition unit 131 acquires the user ID and the user's face image information, the user ID and the user's face image information may be associated and stored in the user information storage unit 121.

Further, the acquisition unit 131 may acquire the user ID for identifying the user and the user's face image information from the authentication device 50 together with the authentication information that the user's face authentication is successful.

Further, the acquisition unit 131 may acquire an RGB image taken in a space that can be photographed by the RGB camera 20 from the RGB camera 20. For example, the acquisition unit 131 may acquire an RGB image taken at a place where the RGB camera 20 is installed from the RGB camera 20 as an example of a space that can be photographed by the RGB camera 20.

Further, the acquisition unit 131 may acquire two or more images having different shooting times from each of the two or more different RGB cameras 20. For example, the acquisition unit 131 may acquire RGB images of moving users from two or more different RGB cameras 20 installed at different places.

Further, the acquisition unit 131 may acquire a thermal image taken in a space that can be photographed by the infrared camera 30 from the infrared camera 30. For example, the acquisition unit 131 may acquire a thermal image taken at a place where the infrared camera 30 is installed from the infrared camera 30 as an example of a space that can be photographed by the infrared camera 30.

Further, the acquisition unit 131 may acquire two or more thermal images having different shooting times from each of the two or more different infrared cameras 30. For example, the acquisition unit 131 may acquire thermal images of moving users from two or more different infrared cameras 30 installed at different locations.

Further, the acquisition unit 131 may acquire environmental information regarding the temperature and humidity detected by the environment sensor 40 installed in a space that cannot be photographed by the photographing device from the environment sensor 40. For example, the acquisition unit 131 may acquire environmental information regarding temperature and humidity detected by the environment sensor 40 installed in a place where the imaging device is not installed, as an example of a space in which imaging is not possible, from the environment sensor 40. .. When the acquisition unit 131 acquires the environment information, the sensor ID that identifies the environment sensor 40, the acquired environment information, and the acquisition time of the environment information may be associated and stored in the environment information storage unit 122.

In addition, the acquisition unit 131 may acquire environmental information regarding temperature and humidity detected by the environment sensor 40 installed in a space that can be photographed by the photographing device from the environment sensor 40. For example, the acquisition unit 131 may acquire environmental information regarding the temperature and humidity detected by the environment sensor 40 installed at the place where the photographing apparatus is installed from the environment sensor 40 as an example of the space in which the photographing apparatus can be photographed. When the acquisition unit 131 acquires the environment information, the sensor ID that identifies the environment sensor 40, the acquired environment information, and the acquisition time of the environment information may be associated and stored in the environment information storage unit 122.

Further, the acquisition unit 131 may acquire information about various machine learning models (hereinafter, may be referred to as "learning model") from an external information processing device. Specifically, as an example of the learning model, the acquisition unit 131 acquires a learning model M1 that outputs a feature vector V1 indicating the characteristics of the RGB image as output information when the RGB image is input as input information. good. For example, the acquisition unit 131 is a learning model M1 learned to estimate an object name (class) appearing in an image from a large number of images (for example, exceeding 10 million images) registered in an image database such as ImageNet. May be obtained. For example, the acquisition unit 131 may acquire the learning model M1 which is a convolutional neural network. For example, the acquisition unit 131 may acquire the learning model M1 which is Renet50. For example, the acquisition unit 131 may acquire a machine learning model M1 that outputs a 2048-dimensional feature vector V1 indicating the characteristics of an RGB image as output information when an RGB image is input as input information. Subsequently, when the learning model M1 is acquired, the acquisition unit 131 may store the information about the learning model M1 and the identification information “M1” that identifies the learning model M1 in association with each other in the storage unit 120.

Further, as an example of the learning model, the acquisition unit 131 may acquire a learning model M2 that outputs a feature vector V2 indicating the characteristics of the thermal image as output information when the thermal image is input as input information. For example, the acquisition unit 131 is learned to estimate the object name (class) reflected in the thermal image from a large number of thermal images (for example, exceeding 10 million images) registered in an image database such as ImageNet. The model M2 may be acquired. For example, the acquisition unit 131 may acquire the learning model M2 which is a convolutional neural network. For example, the acquisition unit 131 may acquire the learning model M2 which is Renet50. For example, the acquisition unit 131 may acquire a machine learning model M2 that outputs a 2048-dimensional feature vector V2 indicating the characteristics of the thermal image as output information when the thermal image is input as input information. Subsequently, when the learning model M2 is acquired, the acquisition unit 131 may store the information about the learning model M2 and the identification information “M2” that identifies the learning model M2 in association with each other in the storage unit 120.

Further, as an example of the learning model, the acquisition unit 131 may acquire the learning model M3 that outputs the feature vector V3 indicating the characteristics of the environmental information as the output information when the environmental information is input as the input information. Specifically, for example, when the information on the temperature and humidity of the space is input as the input information, the acquisition unit 131 uses the feature vector V3 showing the characteristics of the information on the temperature and humidity of the space as the output information. The learning model M3 to be output may be acquired. For example, the acquisition unit 131 may acquire the learning model M3 trained to estimate the feature vector indicating the feature of the environmental information from the environmental information. For example, the acquisition unit 131 may acquire the learning model M3 which is a neural network. For example, the acquisition unit 131 may acquire a machine learning model M3 that outputs a 2048-dimensional feature vector V3 indicating the characteristics of the environmental information as output information when the environmental information is input as input information. Subsequently, when the learning model M3 is acquired, the acquisition unit 131 may store the information about the learning model M3 and the identification information “M3” that identifies the learning model M3 in association with each other in the storage unit 120.

Further, as an example of the learning model, the acquisition unit 131 outputs the comfort of the space for the person as the output information when the feature vector V4 indicating the characteristics of the person located in the space is input as the input information. The estimation model M4 may be acquired. For example, the comfort estimation model M4 learned by the acquisition unit 131 to output the probability that the feature vector V4 is classified into each category indicating comfort when the feature vector V4 is input as input information is output as output information. May be obtained. For example, the acquisition unit 131 determines the probability of being classified in the "cold" category, the probability of being classified in the "hot" category, and the probability of being classified in the "normal" category when the feature vector V4 is input as input information. The comfort estimation model M4 to be output as output information may be acquired.

The category indicating comfort may be any classification and is not limited to the above example. For example, the comfort classification may include other classifications such as "cool" and "warm". Further, the acquisition unit 131 acquires the comfort estimation model M4 learned to output the label of the comfort category in which the feature vector V4 is classified as output information when the feature vector V4 is input as input information. You can do it. For example, when the feature vector V4 is input as input information, the acquisition unit 131 is learned to output the label of the category having the highest probability that the feature vector V4 is classified into each category indicating comfort as output information. The comfort estimation model M4 may be acquired. Subsequently, when the comfort estimation model M4 is acquired, the acquisition unit 131 stores the information about the comfort estimation model M4 and the identification information "M4" that identifies the comfort estimation model M4 in the storage unit 120 in association with each other. good.

(Estimation unit 132)
The estimation unit 132 may specify a user located in the space. Specifically, the estimation unit 132 may specify a user located in a space that can be photographed by the photographing device. The estimation unit 132 may specify a user located at a place where the photographing device is installed as an example of a space that can be photographed by the photographing device. For example, when the estimation unit 132 acquires the authentication information from the authentication device 50, the RGB image taken by the RGB camera 20 and the RGB image taken by the RGB camera 20 before a predetermined time (for example, 5 minutes ago) are RGB. It may be obtained from the camera 20. Subsequently, when the estimation unit 132 acquires the RGB image and the RGB image', the acquired RGB image and the RGB image' are compared, and the person who is not shown in the RGB image'and is shown in the RGB image ( That is, the person newly reflected in the RGB image) may be identified as the user authenticated by the authentication device 50.

Alternatively, the estimation unit 132 may acquire an RGB image taken by the RGB camera 20 from the RGB camera 20. Subsequently, when the estimation unit 132 acquires the RGB image, the face image information stored in the user information storage unit 121 is collated with the RGB image, and the face image information stored in the user information storage unit 121 is used for identification. It may be determined whether or not the user to be used is included in the RGB image.

Further, the estimation unit 132 may specify a user located in a space where photography is not possible by the photographing device. As an example of a space that cannot be photographed by the photographing device, the estimation unit 132 may specify a user located in a place where the photographing device is not installed. More specifically, the estimation unit 132 identifies the authentication information acquired from the authentication device 50 installed at the entrance of the space that cannot be photographed by the photographing device and the user stored in the user information storage unit 121. By collating with possible authentication information, a user who exists in a space that cannot be photographed by the photographing device may be identified. For example, the estimation unit 132 includes face image information acquired from the face recognition device 50, which is a face recognition device installed at the entrance of a space that cannot be photographed by the photographing device, and the user stored in the user information storage unit 121. By collating with the face image information, a user existing in a space that cannot be photographed by the photographing device may be identified.

Further, the estimation unit 132 may generate RGB image information regarding the RGB image. For example, when the RGB image is acquired by the acquisition unit 131, the estimation unit 132 may specify the user's person area R1 among the RGB images acquired by the acquisition unit 131 by the posture estimation process. Subsequently, the estimation unit 132 may generate information about the person area R1 as an example of RGB image information when the person area R1 is specified. For example, the estimation unit 132 may acquire information about the learning model M1 from the storage unit 120 when the person region R1 is specified. Subsequently, when the estimation unit 132 acquires the information regarding the learning model M1, the learning model M1 may be used as an example of the information regarding the person region R1 to generate a feature vector V1 showing the characteristics of the person region R1.

Further, the estimation unit 132 may generate thermal image information regarding the thermal image. Specifically, when the thermal image is acquired by the acquisition unit 131, the estimation unit 132 may specify the user's person area R2 in the thermal image acquired by the acquisition unit 131 by the posture estimation process. For example, the estimation unit 132 superimposes the RGB image captured in the same space and the thermal image, and sets the area of the thermal image overlapping with the person area R1 specified based on the posture estimation process for the RGB image as the person of the thermal image. It may be specified as region R2. Subsequently, when the estimation unit 132 specifies the person area R2, the estimation unit 132 may generate information about the person area R2 as an example of the thermal image information. For example, when the estimation unit 132 specifies the person area R2, the estimation unit 132 may acquire information about the learning model M2 from the storage unit 120. Subsequently, when the estimation unit 132 acquires the information regarding the learning model M2, the learning model M2 may be used as an example of the information regarding the person region R2 to generate a feature vector V2 showing the characteristics of the person region R2.

Further, when the estimation unit 132 generates the RGB image information and the thermal image information, the generated RGB image information and the thermal image information may be associated with the user's face image information and stored in the user information storage unit 121. For example, when the estimation unit 132 generates the feature vector V1 showing the features of the person area R1 and the feature vector V2 showing the features of the person area R2, the generated feature vectors V1 and the feature vector V2 correspond to the face image information of the user. It may be attached and stored in the user information storage unit 121.

Further, the estimation unit 132 may generate a feature vector V3 indicating the features of the environmental information. Specifically, the estimation unit 132 may acquire information about the learning model M3 from the storage unit 120 when the environmental information is acquired by the acquisition unit 131. Subsequently, when the estimation unit 132 acquires the information regarding the learning model M3, the learning model M3 may be used to generate a feature vector V3 indicating the characteristics of the environmental information acquired by the acquisition unit 131. For example, the estimation unit 132 may generate a feature vector V3 indicating the characteristics of the environmental information at the place where the photographing apparatus is installed. Further, the estimation unit 132 may generate a feature vector V3'indicating a feature of environmental information in a place where a photographing device is not installed.

Further, the estimation unit 132 may estimate the comfort of the space for the user by using the comfort estimation model M4. Specifically, the estimation unit 132 may estimate the comfort of the space for the user in a space that cannot be photographed by the photographing device by using the comfort estimation model M4. The estimation unit 132 may estimate the comfort of the space for the user in a place where the photographing device is not installed, as an example of the space that cannot be photographed by the photographing device. More specifically, the estimation unit 132 has a feature vector V1 showing the characteristics of the person area R1 in the place where the RGB camera 20 is installed and a feature vector V2 showing the features of the person area R2 in the place where the infrared camera 30 is installed. May be used to estimate the comfort of the space for the user in a place where the imaging device is not installed. For example, the estimation unit 132 shows the feature vector V1 at the place where the RGB camera 20 is installed, the feature vector V2 at the place where the infrared camera 30 is installed, and the features of the environmental information at the place where the photographing device is not installed. The feature vector V3'may be input to the comfort estimation model M4, and the comfort of the space for the user in the place where the photographing device is not installed may be output as the output information of the comfort estimation model M4.

More specifically, when the estimation unit 132 generates the feature vector V1, the feature vector V2, and the feature vector V3', the feature vector V1 is based on the generated feature vector V1, feature vector V2, and feature vector V3'. , Feature vector V2, and feature map MP1'showing features of feature vector V3'may be generated. Subsequently, when the estimation unit 132 generates the feature map MP1', the estimation unit 132 may generate the feature vector V4'showing the features of the feature map MP1'from the feature map MP1'by the maximum value pooling with respect to the feature map MP1'. For example, the estimation unit 132 may generate a 2048-dimensional feature vector V4'showing the features of the feature map MP1'by maximum value pooling. Subsequently, when the estimation unit 132 generates the feature vector V4', the comfort estimation model M4 is used to estimate the comfort of the space for the user in the place where the photographing device is not installed from the feature vector V4'. You can do it.

For example, the estimation unit 132 may input the feature vector V4'as input information of the comfort estimation model M4. Subsequently, the estimation unit 132 may output the comfort of the space for the user in the place where the photographing device is not installed as the output information of the comfort estimation model M4. For example, when the feature vector V4'is input as input information of the comfort estimation model M4, the estimation unit 132 has the highest probability that the feature vector V4'is classified among the three categories indicating comfort. The label may be output as output information. For example, the estimation unit 132 has a "cold" category indicating that the temperature of the space is low, a "hot" category indicating that the temperature of the space is high, and a space when the feature vector V4'is input as input information. Among the "normal" categories indicating that the temperature of the above is just right, the label of the category with the highest probability that the feature vector V4'is classified may be output as output information.

Further, the estimation unit 132 may estimate the comfort of the space for the user in the space that can be photographed by the photographing apparatus by using the comfort estimation model M4. As an example of the space that can be photographed by the photographing device, the estimation unit 132 may estimate the comfort of the space for the user in the place where the photographing device is installed. For example, the estimation unit 132 shows the feature vector V1 at the place where the RGB camera 20 is installed, the feature vector V2 at the place where the infrared camera 30 is installed, and the features of the environmental information at the place where the photographing device is installed. The feature vector V3 may be input to the comfort estimation model M4, and the comfort of the space for the user in the place where the photographing device is installed may be output as the output information of the comfort estimation model M4.

More specifically, when the estimation unit 132 generates the feature vectors V1 to the feature vector V3, the estimation unit 132 generates the feature map MP1 showing the features of the feature vectors V1 to the feature vector V3 based on the generated feature vectors V1 to the feature vector V3. May be generated. Subsequently, when the feature map MP1 is generated, the estimation unit 132 may generate a feature vector V4 indicating the features of the feature map MP1 from the feature map MP1 by the maximum value pooling with respect to the feature map MP1. For example, the estimation unit 132 may generate a 2048-dimensional feature vector V4 showing the features of the feature map MP1 by maximum value pooling. Subsequently, when the estimation unit 132 generates the feature vector V4, the comfort estimation model M4 may be used to estimate the comfort of the space for the user in the place where the photographing device is installed from the feature vector V4. ..

For example, the estimation unit 132 may input the feature vector V4 as input information of the comfort estimation model M4. Subsequently, the estimation unit 132 may output the comfort of the space for the user in the place where the photographing device is installed as the output information of the comfort estimation model M4. For example, when the feature vector V4 is input as input information of the comfort estimation model M4, the estimation unit 132 displays the label of the category having the highest probability that the feature vector V4 is classified among the three categories indicating comfort. It may be output as output information. For example, the estimation unit 132 includes a "cold" category indicating that the temperature of the space is low, a "hot" category indicating that the temperature of the space is high, and a space, when the feature vector V4 is input as input information. Among the "normal" categories indicating that the temperature is just right, the label of the category with the highest probability that the feature vector V4 is classified may be output as output information.

(Air conditioning control unit 133)
The air conditioning control unit 133 may control the air conditioning of the space based on the comfort of the space estimated by the estimation unit 132. Specifically, the air conditioning control unit 133 may control the air conditioning of the space that cannot be photographed by the photographing device based on the comfort of the space estimated by the estimation unit 132. For example, the air-conditioning control unit 133 may control the air-conditioning device 10 that air-conditions the space where the user is located, based on the comfort of the space estimated by the estimation unit 132. For example, when the comfort of the space estimated by the estimation unit 132 is in the "cold" category, the air conditioning control unit 133 sets the space where the user is located so as to raise the temperature of the space where the user is located. The installed air conditioner 10 may be controlled. Further, when the comfort of the space estimated by the estimation unit 132 is in the "normal" category, the air conditioning control unit 133 is installed in the space where the user is located so as to maintain the temperature of the space where the user is located. The air conditioner 10 may be controlled. Further, when the comfort of the space estimated by the estimation unit 132 is in the "hot" category, the air conditioning control unit 133 sets the space where the user is located so as to lower the temperature of the space where the user is located. The installed air conditioner 10 may be controlled.

Further, the air conditioning control unit 133 may control the air conditioning of the space that can be photographed by the photographing device based on the comfort of the space estimated by the estimation unit 132, as in the space that cannot be photographed by the photographing device.

[1-5. Modification example]
The information processing system 1 according to the first embodiment described above may be implemented in various different forms other than the above embodiment. Therefore, another embodiment of the information processing system 1 will be described below. The same parts as those in the embodiment are designated by the same reference numerals, and the description thereof will be omitted.

[1-5-1. Comfort estimation model]
In the first embodiment, an example has been described in which the estimation unit 132 estimates the comfort of a space for a user in a space that cannot be photographed by an imaging device based only on information about a person area in the image. However, the estimation unit 132 estimates the comfort of the space for the user in a space that cannot be photographed by the photographing device, based on the information about the image (the image including the area other than the person area) before the posture estimation process is performed. You can do it. For example, the estimation unit 132 is a space for a user in a space that cannot be photographed by an imaging device based on an image including an area in which a person is imaged and a space around the person in which the person is imaged. You may estimate the comfort.

More specifically, when the RGB image captured by the user is acquired by the acquisition unit 131, the estimation unit 132 corrects the RGB image based on the environmental information acquired in the space that cannot be captured by the photographing device. You can do it. For example, the estimation unit 132 may correct the RGB image based on the temperature and humidity information acquired by the environment sensor 40 installed in a space that cannot be photographed by the photographing apparatus. That is, the estimation unit 132 may generate an RGB image corrected based on information on temperature and humidity in a space that cannot be photographed by the photographing apparatus.

Subsequently, when the estimation unit 132 generates the corrected RGB image, the user's person area R1'may be specified from the RGB images corrected by the posture estimation process. Subsequently, the estimation unit 132 may generate information about the person area R1'as an example of the corrected RGB image information when the person area R1'is specified. For example, when the estimation unit 132 specifies the person area R1', the estimation unit 132 may acquire information about the learning model M1 from the storage unit 120. Subsequently, when the estimation unit 132 acquires the information regarding the learning model M1, the learning model M1 is used as an example of the information regarding the person region R1 ′ to generate a feature vector V1 ′ indicating the characteristics of the person region R1 ′. good.

Further, when the estimation unit 132 generates the corrected RGB image, the estimation unit 132 may acquire information about the learning model M1 from the storage unit 120. Subsequently, when the estimation unit 132 acquires the information regarding the learning model M1, the estimation unit 132 may generate a feature vector V5 showing the characteristics of the corrected RGB image as an example of the corrected RGB image information. For example, the estimation unit 132 may generate a feature vector V5 showing the characteristics of the corrected RGB image from the corrected RGB image by using the learning model M1.

Further, when the thermal image captured by the user is acquired by the acquisition unit 131, the estimation unit 132 may correct the thermal image based on the environmental information acquired in a space that cannot be photographed by the photographing apparatus. For example, the estimation unit 132 may correct the thermal image based on the temperature and humidity information acquired by the environment sensor 40 installed in a space that cannot be photographed by the photographing apparatus. That is, the estimation unit 132 may generate a thermal image corrected based on information on temperature and humidity in a space that cannot be photographed by the photographing apparatus.

Subsequently, when the estimation unit 132 generates the corrected thermal image, the user's person area R2'may be specified from the corrected thermal image. Subsequently, the estimation unit 132 may generate information about the person area R2'as an example of the corrected thermal image information when the person area R2'is specified. For example, when the estimation unit 132 specifies the person area R2', the estimation unit 132 may acquire information about the learning model M2 from the storage unit 120. Subsequently, when the estimation unit 132 acquires the information regarding the learning model M2, the learning model M2 is used as an example of the information regarding the person region R2'to generate a feature vector V2'indicating the characteristics of the person region R2'. good.

Further, the estimation unit 132 may acquire information about the learning model M2 from the storage unit 120 when the corrected thermal image is generated. Subsequently, when the estimation unit 132 acquires the information regarding the learning model M2, the estimation unit 132 may generate a feature vector V6 indicating the characteristics of the corrected thermal image as an example of the corrected thermal image information. For example, the estimation unit 132 may generate a feature vector V6 indicating the characteristics of the corrected thermal image from the corrected thermal image by using the learning model M2.

Further, the estimation unit 132 may use the learning model M3 to generate a feature vector V3'indicating the characteristics of information regarding temperature and humidity in a space that cannot be photographed by the photographing apparatus.

Further, when the estimation unit 132 generates the feature vector V1'to the feature vector V3'and the feature vector V5 to the feature vector V6, the feature map MP2 based on the feature vector V1'to the feature vector V3'and the feature vector V5 to the feature vector V6. May be generated. Subsequently, the estimation unit 132 may perform a maximum value pooling process on the feature map MP2 to generate a feature vector V7 indicating the features of the feature map MP2 as input information of the comfort estimation model M5.

Further, the acquisition unit 131 obtains a comfort estimation model M5 that outputs the comfort of the space for the user as output information when the feature vector V7 indicating the characteristics of the user located in the space is input as input information. You may get it. For example, the comfort estimation model M5 learned by the acquisition unit 131 to output the probability that the feature vector V7 is classified into each category indicating comfort when the feature vector V7 is input as input information is output as output information. May be obtained. For example, the acquisition unit 131 determines the probability of being classified in the "cold" category, the probability of being classified in the "hot" category, and the probability of being classified in the "normal" category when the feature vector V7 is input as input information. The comfort estimation model M5 to be output as output information may be acquired.

When the estimation unit 132 generates the feature vector V7, the estimation unit 132 may input the feature vector V7 into the comfort estimation model M5 acquired by the acquisition unit 131. Subsequently, the estimation unit 132 may output class information indicating the state of comfort in the space as output information of the comfort estimation model M5.

[1-5-2. Information on the amount of clothes]
Further, in the first embodiment, the user information storage unit 121 has a feature vector V1 showing the characteristics of the user's person area R1 included in the RGB image and the user's face image information as an example of the information regarding the person area. The example of storing in association with and has been described, but the present invention is not limited to this.
Specifically, the user information storage unit 121 may store the information regarding the amount of clothing of the user and the facial image information of the user in association with each other as an example of the information regarding the person area.

More specifically, the acquisition unit 131 is learned to estimate the information on the amount of clothing of the user from the information on the part area of each part of the body of the user specified based on the posture estimation process in the RGB image. The learning model M6 may be acquired. For example, the acquisition unit 131 provides information on the amount of clothing of the user included in any image from any image based on the combination of the image for learning and the information on the amount of clothing of the user included in the image for learning. The learning model M6 for estimating the above may be acquired.

Further, as an example of the information regarding the person area, the estimation unit 132 may estimate the information regarding the amount of clothing of the user based on the information regarding the part area of each part of the body of the user. More specifically, the estimation unit 132 may specify a region of each part of the user's body based on the posture estimation process in the RGB image. Subsequently, the estimation unit 132 may generate a site area image in which the user's site area is cut out for each part by using a program for cutting out each site area when the user's site area is specified. Subsequently, when the estimation unit 132 generates the site region image, the estimation unit 132 may estimate the information regarding the amount of clothing of the user from the information regarding the region region by using the learning model M6 acquired by the acquisition unit 131.

For example, the estimation unit 132 outputs the probability that the combined image is classified into each class when the combined image in which the person region and the site region image are combined is input as input information by using the learning model M6. good. Subsequently, the estimation unit 132 may estimate information on the clothing amount of the person included in the combined image based on the output probability value of each class and the clo value indicating the clothing amount of each class. For example, the estimation unit 132 may calculate a value obtained by adding a value obtained by multiplying the output probability value of each class and the clo value of each class as the estimated clothing amount. The estimation unit 132 may calculate the estimated clothing amount as information regarding the clothing amount of the user. When the estimation unit 132 calculates the estimated clothing amount, the estimated clothing amount of the user and the face image information of the user may be associated and stored in the user information storage unit 121.

[1-5-3. Location information of terminal device]
Further, in the first embodiment, an example in which the user information storage unit 121 stores the user's face image information and the image information in association with each other as an example of the information that can identify the user has been described. Not limited to. Specifically, the user information storage unit 121 may store the position information of the terminal device (smartphone or the like) used by the user in association with the image information as an example of the information that can identify the user. ..

More specifically, the acquisition unit 131 may acquire the user's location information as an example of the information that can identify the user. For example, the acquisition unit 131 may acquire the position information of the terminal device (smartphone or the like) used by the user. For example, the acquisition unit 131 includes GNSS (Global Navigation Satellite System) represented by GPS (Global Positioning System), WiFi (registered trademark) positioning, magnetic positioning, BLE (Bluetooth (registered trademark) Low Energy) positioning, beacon positioning, and the like. May be used to acquire the user's location information. When the acquisition unit 131 acquires the user's position information, the acquisition unit 131 may store the user's position information and the image information in the storage unit 120 in association with each other.

Further, the estimation unit 132 is a space comfort for the user in a space that cannot be photographed by the photographing device based on the environmental information regarding the temperature and humidity detected by the environment sensor located within a predetermined range from the position of the user. You may estimate the temperature. For example, the estimation unit 132 identifies an environment sensor located within a predetermined range from the user's position based on the position information acquired by the acquisition unit 131, and the environment information regarding the temperature and humidity detected by the specified environment sensor. Based on this, the comfort of the space for the user in a space where photography is not possible may be estimated.

As a result, the information processing apparatus 100 provides the comfort of the space for the user in a space where photography is not possible, for example, based on the environmental information regarding the temperature and humidity detected by the nearest environment sensor from the user's current position. Can be estimated. Therefore, the information processing apparatus 100 can accurately estimate the comfort of the space for the user in the space where the image cannot be taken.

[2. Second embodiment]
[2-1. Introduction]
Similar to the information processing device 100 according to the first embodiment, the information processing device 100A according to the second embodiment has an estimated amount of clothing based on image information of a photographing device installed at an entrance / exit of a facility or the like. Information on the amount of metabolism (activity) is used as information for estimating comfort in the facility thereafter. Further, the information processing apparatus 100A updates information on the amount of clothing and the amount of metabolism (activity amount) of the person each time the person is photographed by the photographing device installed in the facility.

Thereby, the information processing apparatus 100A can estimate in advance the comfort of the space in the place where the user is going to move. For example, the information processing apparatus 100A can estimate in advance the comfort of the space of each of the plurality of places where the user may move from now on. Then, the information processing apparatus 100A can provide the user with information on the place where the user can spend the most comfortably based on the estimated comfort of the space. Further, the information processing apparatus 100A can optimally control the state of air conditioning in a specific place in advance before the user moves to the specific place based on the estimated comfort of the space.

[2-2. Information processing device configuration]
Next, the configuration of the information processing apparatus 100A according to the second embodiment will be described with reference to FIG. FIG. 6 is a diagram showing a configuration example of the information processing apparatus 100A according to the second embodiment. As shown in FIG. 6, the information processing apparatus 100A is different from the information processing apparatus 100 in that the providing unit 134A is further provided.

(Providing unit 134A)
The providing unit 134A may provide the user with information on a place where the user can spend comfortably based on the comfort of the space estimated by the estimation unit 132. Specifically, the providing unit 134A may generate information about a place where the user can spend comfortably based on the comfort of the space estimated by the estimation unit 132. Subsequently, when the providing unit 134A generates information about a place where the user can spend comfortably, the providing unit 134A may provide the generated information to the user.

[2-3. Information processing overview]
Next, the outline of the information processing according to the second embodiment will be described with reference to FIGS. 7-8. First, it will be described with reference to FIG. FIG. 7 is a diagram for explaining an outline of information processing according to the second embodiment. Reference numerals P21 to P25 shown in FIG. 7 indicate five locations on the floor F2 to be controlled by the information processing apparatus 100A. In the following, the place indicated by "P2 *" (* = 1 to 5) may be described as "place P2 *" (* = 1 to 5).

In FIG. 7, the information processing device 100A is selected from four locations P22 to P25, which are candidates for the location where the user U1 will move to the user U1 at the location P11 where the photographing device is installed. , Provides information about where the user U1 can spend the most comfortably.

Specifically, the estimation unit 132 is for the user U1 at each of the locations P22 to P25 where the imaging device is not installed, based on the RGB image information and the thermal image information at the location P11 where the imaging device is installed. You may estimate the comfort of the space. The providing unit 134A may compare the comforts of the places P22 to the place P25 estimated by the estimation unit 132, and select the place where the user U1 can spend the most comfortably from the places P22 to the place P25. For example, the provision unit 134A has a label in the "normal" category indicating that the temperature of the space is just right from the locations P22 to P25 estimated by the estimation unit 132 as the place where the user U1 can spend the most comfortably. You may select the output location. When the providing unit 134A selects a place where the user U1 can spend the most comfortably, the providing unit 134A may provide the user U1 with information about the selected place.

Next, it will be described with reference to FIG. FIG. 8 is a diagram for explaining an outline of information processing according to the second embodiment. As shown in FIG. 8, the providing unit 134A may display map information that visually displays a place where a person can spend comfortably on the screen of a terminal device that can be visually recognized by the person.

[3. effect〕
As described above, the information processing apparatus 100 according to the embodiment includes an estimation unit 132. The estimation unit 132 is information about a person area, which is an area where a person is captured, among images taken in a space that can be photographed by a photographing device, and is based on information about a person area specified based on the posture of the person. Therefore, the comfort of the space for a person in a space where photography is impossible is estimated by the photographing device. Further, the estimation unit 132 is a space comfort for a person in a space that cannot be photographed by the infrared camera, based on information about a person area in the thermal image photographed in the space that can be photographed by the infrared camera as a photographing device. To estimate.

In this way, the information processing device 100 substitutes for image information about an image taken in a place near the place where the image pickup device is installed, for example, in order to estimate the comfort of the space in the place where the image pickup device is not installed. To use. As a result, the information processing device 100 can accurately estimate the comfort of the space in a place where the photographing device is not installed. Further, the information processing device 100 can accurately estimate the comfort of the space in a place (toilet, dressing room, etc.) where the photographing device cannot be installed in order to protect the privacy of the user. That is, the information processing device 100 can improve the estimation accuracy of the comfort of the space in a place where the image pickup device cannot take an image. Further, the information processing apparatus 100 can reduce the number of imaging devices such as an RGB camera and an infrared camera installed. Therefore, the information processing apparatus 100 can reduce the cost for accurately estimating the comfort of the space.

Further, the information processing apparatus 100 further includes an acquisition unit 131. The acquisition unit 131 acquires environmental information regarding temperature and humidity detected by an environmental sensor installed in a space where photography is not possible. The estimation unit 132 estimates the comfort of the space for a person in a space where photography is not possible, based on the environmental information regarding temperature and humidity acquired by the acquisition unit 131.

As a result, the information processing apparatus 100 can accurately estimate the environmental elements in the space that cannot be photographed by the photographing device. Therefore, the information processing device 100 can accurately estimate the comfort of the space for the user in a space that cannot be photographed by the photographing device.

Further, the estimation unit 132 estimates the comfort of the space for the person in a space where photography is impossible, based on the environmental information regarding the temperature and humidity detected by the environment sensor located within a predetermined range from the position of the person. Specifically, the acquisition unit 131 acquires the position information of the person. The estimation unit 132 identifies an environment sensor located within a predetermined range from the position of the person based on the position information acquired by the acquisition unit 131, and is based on the environment information regarding the temperature and humidity detected by the specified environment sensor. , Estimate the comfort of the space for a person in a space that cannot be photographed.

As a result, the information processing apparatus 100 provides the comfort of the space for the user in a space where photography is not possible, for example, based on the environmental information regarding the temperature and humidity detected by the nearest environment sensor from the user's current position. Can be estimated. Therefore, the information processing device 100 can accurately estimate the comfort of the space for the user in a space that cannot be photographed by the photographing device.

Further, the information processing apparatus 100 further includes a storage unit 120. The acquisition unit 131 acquires an image from the photographing device. The storage unit 120 stores image information related to an image and information that can identify a person in association with each other.

Thereby, the information processing apparatus 100 can always use the image taken in the space that can be photographed by the photographing device in order to estimate the comfort of the space that cannot be photographed by the photographing device.

Further, the acquisition unit 131 acquires two or more images having different shooting times from each of the two or more different shooting devices. The storage unit 120 stores the image information related to the image having the latest shooting time among the two or more images and the information that can identify a person in association with each other.

Thereby, the information processing apparatus 100 can estimate the comfort of the space for the user based on the latest RGB image information and the thermal image information of the user. Therefore, the information processing apparatus 100 provides information on the user's latest clothing amount and metabolism amount (activity amount) even when the information on the user's clothing amount and metabolism amount (activity amount) changes with the passage of time. Based on this, it is possible to estimate the comfort of the space for the user. Therefore, the information processing device 100 can more accurately estimate the comfort of the space for the user in a space that cannot be photographed by the photographing device.

Further, the storage unit 120 stores information about the person area and information that can identify the person in association with each other as image information. Further, the estimation unit 132 estimates the information regarding the amount of clothing of the person as the information regarding the person area, based on the information regarding the part area of each part of the body of the person. The storage unit 120 stores information about the amount of clothing of a person and information that can identify the person in association with each other.

Thereby, the information processing apparatus 100 can construct a database regarding comfortable environment information for estimating the comfort of each user.

Further, the storage unit 120 stores the item authentication information used in the item authentication process of the person or the biometric authentication information used in the biometric authentication process of the person in association with the image information as the information that can identify the person. .. Further, the estimation unit 132 cannot take a picture by the photographing device by collating the item authentication information or the biometric authentication information acquired from the authentication device with the information that can identify the person stored in the storage unit 120. Identify the person present in the space and estimate the comfort of the space for the identified person.

Thereby, the information processing apparatus 100 can identify the user who is located in the space where the photographing apparatus cannot take a picture.

Further, the information processing device 100 further includes an air conditioning control unit 133. The air conditioning control unit 133 controls the air conditioning of the space that cannot be photographed by the photographing device based on the comfort of the space estimated by the estimation unit 132.

As a result, the information processing device 100 can comfortably control the air conditioning of the space that cannot be photographed by the photographing device.

Further, the information processing apparatus 100A further includes a providing unit 134A. The providing unit 134A provides the person with information about a place where the person can spend comfortably based on the comfort of the space estimated by the estimation unit 132.

Thereby, the information processing apparatus 100A can provide the user with information on the place where the user can spend the most comfortably based on the estimated comfort of the space. Further, the information processing apparatus 100A can optimally control the state of air conditioning in a specific place in advance before the user moves to the specific place based on the estimated comfort of the space.

Further, the providing unit 134A displays the map information that visually displays the place where the person can spend comfortably on the screen of the terminal device that can be visually recognized by the person.

Thereby, the information processing apparatus 100 can improve the usability of the user who receives the information about the place where the user can spend the most comfortably.

[4. Hardware configuration]
Further, the information processing apparatus 100 according to the first embodiment or the information processing apparatus 100A according to the second embodiment described above is realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100 or the information processing device 100A. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from another device via a predetermined communication network and sends the data to the CPU 1100, and transmits the data generated by the CPU 1100 to the other device via the predetermined communication network.

The CPU 1100 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

For example, when the computer 1000 functions as the information processing device 100 according to the first embodiment or the information processing device 100A according to the second embodiment, the CPU 1100 of the computer 1000 executes a program loaded on the RAM 1200. Thereby, the function of the control unit 130 or the control unit 130A is realized. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be acquired from another device via a predetermined communication network.

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other modified forms.

[5. others〕
Further, among the processes described in the above-described embodiments and modifications, all or part of the processes described as being automatically performed can be manually performed, or are described as being manually performed. It is also possible to automatically perform all or part of the processed processing by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically distributed in any unit according to various loads and usage conditions. Can be integrated and configured.

Further, the above-described embodiments and modifications can be appropriately combined as long as the processing contents do not contradict each other.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the estimation unit can be read as an estimation means or an estimation circuit.

1 Information processing system 10 Air conditioning device 20 RGB camera 30 Infrared camera 40 Environment sensor 50 Authentication device 100 Information processing device 110 Communication unit 120 Storage unit 121 User information storage unit 122 Environmental information storage unit 130 Control unit 131 Acquisition unit 132 Estimating unit 133 Air conditioning control unit 1A Information processing system 100A Information processing device 130A Control unit 134A Providing unit

Claims (17)

The shooting is based on the information about the person area, which is the area where the person is captured in the image taken in the space that can be taken by the shooting device, and is based on the information about the person area specified based on the posture of the person. An estimation unit that estimates the comfort of the space for the person in a space that cannot be photographed by the device.
Information processing device equipped with. The estimation unit
As the photographing device, the comfort of the space for the person in the space that cannot be photographed by the infrared camera is estimated based on the information about the person area in the thermal image photographed in the space that can be photographed by the infrared camera. ,
The information processing apparatus according to claim 1. Further equipped with an acquisition unit for acquiring environmental information regarding temperature and humidity detected by an environmental sensor installed in the space where photography is not possible.
The estimation unit
Based on the temperature and humidity environmental information acquired by the acquisition unit, the comfort of the space for the person in the non-photographable space is estimated.
The information processing apparatus according to claim 1 or 2. The estimation unit
Based on the temperature and humidity environmental information detected by the environment sensor located within a predetermined range from the position of the person, the comfort of the space for the person in the non-photographable space is estimated.
The information processing apparatus according to claim 3. The acquisition unit
Acquire the location information of the person and
The estimation unit
The environment sensor located within a predetermined range from the position of the person is specified based on the position information acquired by the acquisition unit, and the shooting is not performed based on the environment information regarding the temperature and humidity detected by the specified environment sensor. Estimating the comfort of the space for the person in a possible space,
The information processing apparatus according to claim 4. An acquisition unit that acquires the image from the photographing device, and
A storage unit that stores image information related to the image and information that can identify the person in association with each other.
The information processing apparatus according to any one of claims 1 to 5. The acquisition unit
Two or more images having different shooting times are acquired from each of the two or more different photographing devices, and the images are obtained from each of the two or more different photographing devices.
The storage unit is
Among the two or more images, the image information about the image having the latest shooting time and the information that can identify the person are stored in association with each other.
The information processing apparatus according to claim 6. The storage unit is
As the image information, information about the person area and information that can identify the person are stored in association with each other.
The information processing apparatus according to claim 6 or 7. The estimation unit
As the information about the person area, the information about the clothing amount of the person is estimated based on the information about the part area of each part of the body of the person.
The storage unit is
Information about the amount of clothing of the person and information that can identify the person are stored in association with each other.
The information processing apparatus according to any one of claims 6 to 8. The storage unit is
As the information that can identify the person, the item authentication information used for the item authentication process of the person or the biometric authentication information used for the biometric authentication process of the person is stored in association with the image information.
The information processing apparatus according to any one of claims 6 to 9. The estimation unit
By collating the item authentication information or biometric authentication information acquired from the authentication device with the information stored in the storage unit that can identify the person, a person existing in a space that cannot be photographed by the photographing device. And estimate the comfort of the space for the identified person,
The information processing apparatus according to any one of claims 6 to 10. An air conditioning control unit that controls air conditioning in a space that cannot be photographed by the imaging device based on the comfort of the space estimated by the estimation unit.
The information processing apparatus according to any one of claims 1 to 11. A provider that provides the person with information about where the person can spend comfortably, based on the comfort of the space estimated by the estimate.
The information processing apparatus according to any one of claims 1 to 12, further comprising. The providing part
The map information that visually displays the place where the person can spend comfortably is displayed on the screen of the terminal device that can be visually recognized by the person.
The information processing apparatus according to claim 13. The shooting is based on the information about the person area, which is the area where the person is captured in the image taken in the space that can be taken by the shooting device, and is based on the information about the person area specified based on the posture of the person. An estimation procedure that estimates the comfort of the space for the person in a space that cannot be photographed by the device,
An information processing program characterized by having a computer execute. An information processing system that includes a photographing device and an information processing device.
The photographing device captures an image including a person in a space that can be captured by the photographing device.
The information processing device is
Information about a person area, which is an area in which a person is captured in the image, and based on information about a person area specified based on the posture of the person, the person in a space that cannot be photographed by the photographing apparatus. Estimate the comfort of the space for
Information processing system. The information processing system further includes an environment sensor.
The environment sensor detects the temperature and humidity of a space that cannot be photographed by the photographing device, and detects the temperature and humidity of the space.
The information processing device is
Estimating the comfort of the space for the person based on the environmental information regarding the temperature and humidity of the space that cannot be photographed.
The information processing system according to claim 16.

Images