JPH06117836A - Image processing apparatus, controller of air conditioner, and applied equipment using the apparatus - Google Patents

Abstract

PURPOSE:To detect personal information and detailed environmental information in a detecting area by providing a sensor section having an infrared image inputting means and image processing section which processes a thermal image having the detected temperature of the sensor section as an image element value. CONSTITUTION:A two-dimensional infrared sensor 11 inputs the heat distribution on the outside (in a room) 4 to an image processing section 2 as a thermal picture 6. A human area detecting section 13 segments a human body part from the image data based on the segmenting temperature of the human body part, calculates a human image element block and its representative points, and detects a feature amount for detecting the position of a person or discriminating the posture of the person from the calculated block and representative points and the surrounding environment of the representative points. An image information detecting section 3 inputs each feature amount 7 and detects the position, posture, etc., of the person. The detection is performed by using a posture detecting means 18 and foot position detecting means 17 respectively constituted of different neural networks. The means 17 selects the output from a neural network for presuming the foot position of the person at every posture based on the output of the means 18 in accordance with the kind of posture (standing, sitting, etc.) of the person.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The image processing apparatus of the present invention mainly relates to an image processing apparatus for detecting information on a person or environment from a thermal image obtained from an infrared sensor. It can be used in many industrial fields such as equipment and intelligent building systems (IBS). Further, the control device for an air conditioner of the present invention can be used for an air conditioner that uses information on a person or environment obtained mainly from an infrared sensor as a control index.

[0002]

BACKGROUND OF THE INVENTION A first field to be solved by the present invention relates to equipment control using a thermal image of an infrared sensor and human information obtained from the infrared image. Conventionally, there has been a device that detects the position of a person and controls the air conditioning by using an infrared sensor. For example, Japanese Patent Laid-Open No. 2-143047
The configuration described in Japanese Patent Publication is known. The configuration will be briefly described below. The room is divided into a plurality of areas, and an infrared sensor for detecting infrared rays in each area is provided to detect a person in each area and the temperature of the area corresponding to the floor surface. Then, based on the relationship between the detected position of the person, the temperature of the floor near the person, and the room temperature, the set temperature is controlled so that the room temperature reaches the set temperature in the normal time, and the wind direction is the result of detecting the position of the person. Change based on If the room temperature is within the set temperature range and the difference between the floor surface temperature and the room temperature is large, the floor surface is blown to warm the floor surface. By this,
The control corresponding to the area where the person is located and the control for reducing the discomfort when the temperature reaches the set temperature but there is cold around the person's feet are performed.

In addition to the air conditioners, accurate information about human beings is required in surveillance systems and the like. A method of detecting information about human beings is to install a visible surveillance camera for human monitoring or to use video. It is common to record. Although there are monitoring systems that use image processing, detection is often difficult due to problems such as illuminance and color. In addition, there are also surveillance systems that use infrared cameras, but they generally require a large number of detection elements or a cooling device to obtain a high-resolution thermal image, which requires complicated processing and high cost. Is required.

A second field to be solved by the present invention relates to an air-conditioning control index. Regarding the air-conditioning control index, in recent years, PMV (ISO 7730) that considers not only the temperature index but also factors related to comfort other than temperature has been considered.
-1984) and the like relating to air conditioning control using the degree of comfort as a control index. For example, the configuration described in the embodiment of Japanese Patent Laid-Open No. 4-84055 is known. The configuration will be described below. Air temperature, radiant temperature, humidity, and airflow are measured as ambient environment side elements, and human momentum and clothing amount are estimated as human side elements, and control is performed based on the human side elements and environment side elements. The PMV, which is a comfort index, is calculated from the above elements, and control is performed based on the calculated PMV. In this case, the method of deriving the input elements is to measure all the elements on the ambient environment side with a sensor that measures each element, which leads to an increase in cost in commercialization,
In addition, since there are many factors that are difficult to measure in terms of measurement method, PM can be measured using a neural network from easily measurable information.
Guess V. In other words, using a radiation sensor or an airflow sensor for PMV measurement leads to an increase in the cost of the product, and actually arranging the sensor indoors
Have difficulty. Further, even if these plural elements can be detected or estimated, complicated calculation is required to calculate PMV. In order to improve these points, six elements of outside temperature, room temperature, temporal change of room temperature, air volume, set temperature, and person's position, which are relatively easy to measure, are input, and the PMV at that time is used as teacher data in advance. A neural network that outputs PMVs to the six types of inputs is configured by learning the network, PMVs are estimated using this neural network, and the estimated PMVs are used as an air-conditioning control index. In this case, since the teacher data used for learning has many cases, the activity amount and the clothing amount use fixed values of a plurality of combinations.

A third area to be solved by the present invention is:
It is related to air conditioning when not at home. Conventionally, a person detection sensor is provided, or an absent button is provided in the air conditioner operation means, and when an air conditioner in which a person is absent is judged, control is performed to keep the room temperature at a predetermined temperature level to a minimum. There was something to do. As a result, it was possible to prevent the building frame from being overheated when it was absent and from overheating when it was being cooled, which led to a reduction in discomfort when entering the room. In addition, since there is an uncomfortable time until the set temperature is reached from the natural room temperature or the minimum temperature level when the user is absent or sleeping, some timers use the set temperature when entering the room or when waking up. It was

[0006]

However, the control device of the air conditioner using the infrared thermal image shown in the prior art is effective for detecting the direction in which a person is present, but the comfort of each individual person in the room is reduced. It has been difficult to detect accurate information regarding the position of each individual, the posture determination of each individual, and the like in order to accurately calculate the degree.

Further, since the measurement range is only fixed in advance, if the interior of the room has a different form from the expected room, the wall may be erroneously recognized as the floor, and the accurate floor temperature cannot be detected. There was a problem.

That is, these problems are roughly divided into the following two problems. (1) Image resolution is low (2) Individual or indoor environment information cannot be accurately detected (1) is a problem caused by the resolution of the image reading sensor. If the image resolution is low, it is difficult to accurately detect personal and indoor environment information. Therefore, in the present invention, it is desirable that the image has such a resolution that one person in the room can detect it as a group of a plurality of pixels.

Next, with respect to (2), even in a high-resolution image in which (1) does not pose a problem, personal information is accurately detected, and a control device based on this information is not used, which causes a problem. Is. In the present invention, (2) will be raised as a major problem of the conventional method.

With respect to the control device for an air conditioner shown in the second field of the prior art, the method of estimating the comfort level using a neural network is to estimate the average comfort level of the entire room. It was not a control index considering the individual comfort level. As an input element for comfort level calculation, a method for detecting airflow, radiation, etc. may be arranged with a sensor that measures each element, but multiple sensors are arranged indoors in an actual home. Is difficult. Further, it is conceivable to install a sensor for measuring these ambient environment side elements on a remote control or the like, but in that case, measurement of only one environmental element in the room is possible. As for human-side elements, as already used in some products, it is also possible to detect the amount of activity that is the human-side element from the number of times the peak value of the output of the infrared sensor is detected. However, for example, it is estimated that the activity is high even in a situation where 5 people are in the room, 4 are stationary, and 1 person is moving around the room.
There is a problem in the accuracy of detecting the amount of activity for each individual. It is also possible to estimate the amount of clothing from solar radiation, calendar, outside temperature, etc., but since it is not actually measuring the human body, when it is far from the amount of clothing of the person actually staying in the room There was also a problem in terms of accuracy.

The air conditioner control device shown in the third field of the prior art has been controlled to maintain a certain minimum temperature when it is absent, but it still reaches the set temperature when entering or waking up. There was a problem that it took time until it was uncomfortable. Also, regarding the minimum temperature level to be maintained, in the case of heating, there is a problem that if it is too high, the electricity bill is uneconomical, and if it is too low, it takes a considerable time to reach the set temperature. On the other hand, with respect to the timer, there is a problem of time setting and resetting of the timer by the time input format, there are few changes in the time setting, and there is a problem that the timer is not used much except when the user is awakened.

In view of the above-mentioned accuracy of human body position detection, input element detection accuracy for comfort level calculation, and inconvenience of a timer set, the first object of the present invention is to: Detecting human personal information and detailed environmental information in the detection area using an infrared thermal image, and a second purpose is to detect air information based on the human information in the detection area and the detailed environmental information in the room. The third purpose is to determine the fine control index of the harmony machine, and to eliminate the timer operation.

[0013]

In order to achieve the above-mentioned object, the technical solution of the present invention relates to an image processing apparatus, an infrared image (hereinafter, referred to as an infrared image) for measuring the temperature of at least human and environment in a detection area. Called a thermal image) A sensor unit having input means, and a thermal image having a pixel value of the detected temperature output from the sensor unit is processed to extract the feature amount of the image used in the image information detection unit in the next stage. An image processing unit and a neural network or a pattern recognition mechanism are included in the constituent elements, and based on the signal from the image processing unit, a human personal information signal or an environmental information signal equivalent to or more detailed than the output signal from the sensor unit. It has an image information detection unit for estimating and outputting by the neural network or the pattern recognition mechanism.

Further, the image processing unit includes a human region correction unit for complementing and erasing the human pixel block obtained by the human region detection unit at a pixel level, and a representative of the human pixel blocks obtained by the human region detection unit. A cumulative processing unit that accumulates points for a predetermined period, and at least a floor and a wall, or an indoor representative point that is a boundary between the walls, is determined from the cumulative period of the representative points of the human pixel block obtained from the cumulative processing unit, It is also possible to add an indoor part determination unit that associates each pixel on the thermal image with the floor and wall of the room. .

Further, in addition to the above-mentioned structure, a console section for inputting environmental information in a detection area that cannot be detected from an infrared image and a visible image detecting section (camera) for detecting a visible image for direct monitoring by a person are equipped. It is a system that makes effective use of the results of thermal image processing.

Further, in order to achieve the second object, in regard to the control device of the air conditioner, the above-mentioned image processing device is included in the constituent elements, and further, a sensor section for detecting temperature, humidity and the like, and the above-mentioned image processing. It has a control index determination unit that determines a control index for performing air conditioning from a personal information signal obtained from the device and an environmental information signal, and an air conditioning control unit that controls the air conditioning equipment. The control index determination unit is an activity amount detection unit that detects an activity amount representing an individual or a room from the personal information output from the personal information extraction unit of the image processing device, the personal information, and the environmental information extraction. It has a comfort level detection unit that detects individual or representative comfort level from the environmental information obtained from the air conditioner, the wind direction / air volume information obtained from the air conditioner, and the activity amount obtained from the activity amount detection unit. Therefore, the control target value of the air conditioning control unit is output from the comfort level.

Further, a life scene estimating unit for estimating a life scene from the activity amount obtained from the activity amount detecting unit, the time, and the environmental information output from the environmental information extracting unit is added to the control index determining unit. Estimate transitional life scenes in which comfort level does not serve as a control index and stable life scenes in which the occupant stays in the room for a specified period of time and comfort level can be used as a control index. A predetermined predetermined control index is selected for each comfort level and each scene.

Further, in order to achieve the third object, the occupancy data storage means, the occupancy pattern classification means, and the base temperature which is the minimum maintenance temperature level during the absence time are obtained from the occupancy pattern classification means. It has an absent control index determination unit and a calendar that are configured of a base temperature conversion unit that is determined according to the usage status of the room.

[0019]

The image processing apparatus of the present invention has the following structure.
First, the infrared image input means of the sensor unit inputs a thermal image. The input image data is cut out by the human region detection unit based on the cutout temperature of the human body part, and these human pixel blocks and their representative points are calculated. Next, for example, regarding a method of detecting the position and orientation of a human being, which is one of the signals detected by the image processing apparatus, the human pixel block and its representative point The feature amount for position detection and the feature amount for posture determination are detected. Next, the image information detection unit detects the position and posture of a person by inputting each of the feature amounts described above. The position and the posture of the person are detected by the posture detecting unit and the foot position detecting unit which are configured by different neural networks. Further, the foot position detecting means makes a distinction according to the types of postures, for example, according to the number of postures to be discriminated such as standing posture and sitting posture, and from the neural network that estimates the foot position for each posture. The output is selected and selected according to the output from the attitude detecting means. By performing such processing, more accurate position detection becomes possible.

In an actual home, there may be an obstacle such as furniture between the human body to be detected and the sensor. In that case, it is erroneous that the human body is separated and a plurality of people are present in the room. The human area correction unit searches around the pixels cut out as a human pixel block, complements the pixels, and erases pixels considered to be noise. Furthermore, in order to remove stationary heat-generating objects such as a television, the accumulating unit accumulates the representative points of the cut out human pixel blocks for a predetermined period, and when only certain pixels exceed a preset frequency threshold, the It is assumed that there is a heat-generating object, the position on the thermal image is stored, and if the representative point of the human pixel block coincides with the position of the stationary heat-generating object in subsequent detection of the human area, the human pixel block is treated as the human body. Do no processing. This enables highly accurate detection of the human body. Further, the accumulating unit accumulates the representative points of the human pixel block for a period longer than the accumulation period used for detecting the stationary heat-generating object, and the three points of the wall surface and the floor surface excluding the wall surface where the air conditioner is installed The representative point that specifies the boundary is detected, and the indoor part determination unit associates the correspondence between the floor surface and the wall surfaces in the three directions and the thermal image based on the representative point. This allows the environment information extraction unit to detect the temperature of each part of the floor and the wall in the three directions even in rooms of different sizes, and by using the foot position information of the person, radiation with high accuracy can be obtained for each individual. The temperature can be detected.

Further, since it is difficult to detect environmental information in the detection area of the input means of the thermal image, for example, structural information such as the size of the room or the position of the wall surface relative to the sensor in the thermal image from the thermal image, the user Can be directly input from the console unit and used as auxiliary information for accurate position detection and posture detection.

Next, the image processing apparatus of the present invention is a system using not only thermal image processing but also a visible image input apparatus (camera) so that the human-related information obtained from the thermal image can be visualized. It can be communicated on the image to provide the user with the location of the person in the visible image.

In the opposite case, it is possible to detect the position of the person on the thermal image by using the information of the person obtained from the visible image, for example, the motion information, and the processing of the thermal image and the visible image can be performed. The performance of the neural network in the image information detection unit, which is being performed, can be adaptively learned by using mutual information as teachers.

Furthermore, by using a plurality of image processing apparatuses in which the thermal image processing and the visible image processing according to the present invention are combined, it is possible to more accurately detect human information in the area where the plurality of apparatuses are installed. Although it is difficult for one image processing apparatus to accurately separate overlapping persons, this problem can be solved by using a plurality of apparatuses. In addition, people can be tracked in a large area.

Next, regarding the control device of the air conditioner, according to the present invention, with this configuration, first, the image processing device detects human and environment information in the detection area. In the control index determination unit, for example, when the comfort level is used as the control index, the individual information based on the personal information extracted by the personal information extraction unit of the image processing device and the environmental information calculated by the environmental information extraction unit The comfort level is calculated, and when there are a plurality of occupants, a control index for the wind direction, the air volume, and the comfort level is calculated so that the occupants having a low comfort level are preferentially made comfortable. The air conditioning control unit controls the compressor frequency, controls the vertical and horizontal flaps, and controls the indoor unit blowing fan, for example, based on the instruction from the control index determination unit.

Although the control index determination unit has shown an example in which the comfort level is used above, by adding the life scene estimation unit, the comfort level such as sleeping, waking up, entering the room, leaving the room is not a control index. Distinctive scenes and stable scenes where the degree of comfort after a person enters the room is the control index,
Depending on the scene, the transition scene uses a predetermined control index, and the stable scene uses the calculated comfort level as a control index. In addition, when only the scene estimation unit is used as a control index, in addition to estimating transient scenes, stable scenes are further cleaned according to the number of people and the amount of activity representative of the room calculated from the individual information. , Scenes such as relaxation and the like, and a predetermined control index is used for each scene.

Further, with this configuration, first, at a fixed time interval, the presence / absence of the person in the room is detected by the number-of-people detection unit as 1 when the person is present and 0 when the person is not present. Since the data of 0 and 1 change greatly when a person goes in and out of the room, 10 minutes is set as a unit, and for example, correction is made to be 0 only when 0 is continued for 10 minutes or more. The occupancy data storage means stores the supplemented occupancy data for one day for a certain period from the current day. The occupancy pattern classification means collects occupancy data for the days with the same attributes of the current day and weekday based on the information of the attributes of the days such as weekdays, holidays, and days of the week from the calendar, and determines the occupancy rate by time (occupancy pattern and Call). The base temperature conversion means calculates the value of the base temperature based on this occupancy pattern. In the air conditioning control means,
With the output of the person detection means, the set temperature is set as the target value while the person is in the room, and the base temperature value at each time from the current time is set as the target value when the person is not present. Switch the target value to. As a result, since the indoor environment is on standby before a person enters the room, it is excellent in comfort, and since the base temperature is low during times when there is a lot of absence, it is possible to realize economical air conditioning.

[0028]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing apparatus according to the first embodiment of the present invention will be described below.

FIG. 1 is a block diagram of the image processing apparatus of this embodiment. A sensor unit 1 measures a thermal image. An image processing unit 2 detects a human region and a feature amount from the thermal image. An image information detection unit 3 estimates the posture and foot position of a person from the thermal image. The image information detection unit 3 is composed of a plurality of neural networks and the like.

The basic flow of information in FIG. 1 is shown. Outside world 4
Is a region where a person is present, and is scanned by the two-dimensional infrared sensor 11 included in the sensor unit 1 to detect the temperature of an area detectable as thermal image information. It may also refer to a room as an area where people live. The heat distribution in the external environment 4 is scanned by the two-dimensional infrared sensor 11 and is input to the image processing unit 2 as a thermal image 6. Since the entire heat distribution including the person in the scanning area becomes an image in the thermal image 6, the image processing unit 2 detects the area of the person. The feature amount data of the person's region to be transferred from this person's region to the subsequent image information detecting unit 3 is created by the image processing unit 2, and is input to the image information detecting unit 3 as feature amount data 7. The image information detection unit 3 estimates a personal information signal 8 such as a posture determination and a foot position from the feature amount data 7.

The operation of the air conditioner incorporating the image processing apparatus configured as described above will be described. First, for example, a pyroelectric one-dimensional infrared sensor, which is installed in the main body of a wall-mounted air conditioner and in which a plurality of elements are arranged in parallel in the vertical direction, scans the room in the frontal direction at a sampling timing for a fixed time. Using the two-dimensional infrared sensor 11 configured as described above, a two-dimensional thermal image of a person in the room is obtained. FIG. 2A is an example of the measured thermal image in the first embodiment. There are three people in a quadrangular room, and each person has a different posture. Reference numeral 21 is an image block of a person in a lying position, 22 is an image block of a person in a standing position, and 23 is an image block of a person in a chair sitting position. 24 is an image block on the floor surface, 25 is an image block on the left wall surface, 26 is an image block on the back wall surface,
Reference numeral 27 is an image block on the right wall surface. FIG. 2B is a plan view of the room at this time. 28 is a person in a lying position, 29
Is a standing person, 30 is a chair sitting person, 31 is a floor,
32 is a left wall, 33 is a back wall, 34 is a right wall, 35 is an air conditioner having the image processing device as a constituent element, 36 is an infrared two-dimensional sensor installed in the air conditioner 35, and 37 is an infrared ray. The scanning direction and the detection range of the two-dimensional sensor 36. In this way, by scanning the two-dimensional infrared sensor 36 in the front direction, a distant person is detected in the upper part of the thermal image, and a near person is detected in the lower part. Also, most of the floors and walls are detected on the thermal image except for the wall of the air conditioner mounting surface.

Next, a detailed description of FIG. 1 will be given. First, the image processing unit 2 will be described. The thermal image detected by the two-dimensional infrared sensor 11 is used by the image storage unit 1 for the subsequent processing.
Remember once in 2. Next, in the human area detection unit 13, FIG.
With the configuration shown in (1), a human area is detected. Conventionally, when detecting a human region from thermal image data, assuming that there is no heat generation other than human beings in the room and the indoor temperature of the non-human beings is 25 ° C. or lower, the temperature range of 26 ° C. to 34 ° C. The area of the human body is detected by detecting pixels. However, with the above-described configuration, it may not be possible to accurately cut out a human from a two-dimensional thermal image. For example, the temperature distribution in the room may vary depending on the floor and each wall surface.
There is also a method of detecting a person from the boundary line of the person without using a temperature range for cutting out the person, but accurate detection is difficult if a good edge cannot be detected.

Therefore, in this embodiment, the thermal image stored in the image storage unit 12 is divided into M (M is a positive integer) to divide into regions (hereinafter, each region will be referred to as a block), and the thermal image The human cutout temperature range is determined for each block to calculate the human position, and the human cutout temperature is calculated in conjunction with the temperature distribution in the room so that human detection can be easily calculated. I chose

Further, it is possible to effectively use the processing that easily catches a heat-generating object such as a human being or a living thing, which is a feature of the thermal image.
Also, when considering a human as a heat-generating object, the area of the human becomes convex when the ambient temperature is lower than the ambient temperature,
It becomes concave when the ambient temperature is low. Therefore, if the adjacent pixels of the thermal image are G (n) and G (n + 1), the difference is calculated as D (n) = (G (n) -G (n-1)) / 2, Absolute value of difference D (n) (ABS (D
(n)) ABS (D (n))> TH Detects when there is more than the appropriate threshold value TH as the boundary of the human area, and cuts out the human area based on the temperature inside and outside the boundary. There is also a method. When the entire thermal image contains a lot of noise, a polynomial smoothing method is used as one of the well-known moving average methods in image processing as a method for capturing large irregularities in the thermal image while removing noise. There is also a method of finding unevenness corresponding to a person in a pixel and cutting out a region based on the temperature of the uneven part. This method is published in "Waveform data processing for scientific measurement" published by CQ Publishing Co., Ltd.

Further, as a method of connecting a pixel of which the temperature is close to that of each pixel of a thermal image to perform a human area, a simple area expansion is used as an area dividing method shown in "Image Analysis Handbook" published by The University of Tokyo Press. There is also a method for detecting a region of the heat distribution on the thermal image plane by using the method or the iterative region expansion method and detecting a region having a size and temperature distribution that can be judged as a human region in the region.

An example of the configuration of the human area detecting section 13 will be described with reference to FIG. In FIG. 3, reference numeral 40 is a region-division representative temperature calculating means for dividing the thermal image into M blocks and calculating an average temperature of each block, and 41 is a cutout temperature for determining a human cutout temperature from the average temperature of the blocks. Determining means, 42 is a human detecting means, and 43 is a representative point calculating means for calculating a representative point of a human area.

The thermal image stored in the image storage unit 12 is divided into M blocks by the area division representative temperature calculating means 40, and the average temperature of each block is calculated. The calculated average temperature of the blocks is passed to the cut-out temperature determining means 41 that determines the cut-out temperature range of a human, and the cut-out temperature range of each block is determined. Here, the reason for dividing into a plurality of M blocks is that it is difficult to accurately cut out the human body portion at a uniform cutting temperature such as when the wall is partially warmed or clothes of the human body are cold. Therefore, by dividing the thermal image into M pieces, it is possible to cut out in consideration of the background temperature. Next, the cut-out temperature range and the thermal image are input to the human detection unit 42 for detecting the human region, and the human pixel block indicating the human region is detected. Representative point calculating means 43
Then, using a human pixel block as an input, a representative point representative of the block is detected, for example, a temperature-weighted center-of-gravity point pixel or a representative point such as a lowest point pixel corresponding to a foot position. In addition, the coordinates of the pixel closest to the floor surface or the coordinates of the pixel farthest from the floor surface may be used as the representative point so that the number of calculations is small. Further, the position of the Y coordinate of the center of gravity and the position of X closest to the floor surface can be set as the position of the representative point. By knowing this representative point, it is possible to roughly determine where humans are located and how many people there are.

FIG. 4 shows an example in which the thermal image data of the present invention is divided into eight blocks and the average temperature of each block is calculated. Region division representative temperature calculation means 40 for each block
Then, the average temperatures T1 to T8 are calculated. For example, when the value of the average temperature TN of the block N is 20 ° C <TN <22 ° C, the cut-out temperature range of the person in the block N is 25
The cut-out temperature of a person is determined for each block, such as ℃ to 34 ℃. In other words, human exposed skin temperature is 36
The temperature is between ℃ and 37 ℃, but when the clothes, the background and the human body part are included in one pixel, they are averaged and the detected temperature becomes low. Become. In the present invention, an appropriate human cutout temperature range is determined based on the average temperature value of each block and the position of the block in the entire image.

On the other hand, in the above configuration, in order to correctly extract the human region, it is necessary to set the temperature range in which it is determined that the pixel includes the human body for each infrared element.
When the number of elements is large, it is difficult to set a temperature range for each detection element in manufacturing the device, or an object having the same temperature range as a human, such as a television or a heater, is mistaken for a human. Also had. Therefore, there is a method of determining an appropriate human cutout temperature range based on continuous images. FIG. 5 shows the configuration of the human area detection unit 13 using such a method. The difference from the configuration of FIG. 3 is that instead of the area division representative temperature calculating means 40, a difference image creating means 50, a moving object image creating means 51, and a frequency distribution creating means 52.
Is provided, and the following is the same as the configuration of FIG. The operation will be described below with reference to FIG.

With this configuration, the image storage unit 12 continuously stores a plurality of thermal images. The difference image creating means 50 creates a difference image between the plurality of stored continuous thermal images. The moving object image creating means 51 creates an moving object image from a plurality of thermal images and the detected difference images.
Next, in the frequency distribution creating means 52, since the moving object image creating means 51 obtains the pixel blocks of the image corresponding to the moving object among the pixel values of the plurality of thermal images, the frequency values of the temperature values of these pixel blocks are obtained. Create a distribution. The cut-out temperature determining means 53 determines a cut-out temperature range in which a human region can be detected, using an appropriate threshold value set inside and the frequency distribution obtained from the frequency distribution creating means 52. Then, thereafter, the human detection unit 42 and the representative point calculation unit 43 shown in FIG. 3 calculate the human pixel block and its representative point.

Next, a detailed processing procedure will be described with reference to FIG. The image stored in the image storage unit 12 is an image obtained by continuously reading the temperature distribution in the room as a thermal image. In the present embodiment, the continuous measured thermal images are referred to as image A and image B, respectively.

First, the images A and B are read in order to form images. The image is represented by P (x, y) of pixels existing at coordinates (x, y). Image A and image B
From the difference image creating means 50, a difference image is created. In this case, the threshold value h may be used when the image contains a noise component. That is, the coordinates (x,
When the pixel value in y) is T1 (x, y), the pixel value at the same coordinate in the image B is T2 (x, y), the threshold value is h, and an absolutely impossible pixel value is TN, If | T2 (x, y) -T1 (x, y) | <h, the value dT of the pixel at the same coordinates in the difference image is dT (x, y) = TN | T2 (x, y) -T1 ( x, y) | ≧ h, dT (x, y) = T2 (x, y) -T1 (x, y) (| x | indicates that the absolute value of x is determined) A difference image can be created by performing this on all the pixels inside. FIG. 6 shows an example of creating a difference image from the images A and B. By taking the difference between the two images, it can be seen that the area with a positive value is the area after the movement and the negative area is the area before the movement. For TN, for example, if the temperature value is measured in degrees Celsius, TN = -1
Absolutely less than 0 degrees like 000 or TN =
A very high temperature such as 10,000 may be used. Each pixel in the temperature difference image indicates that the pixel has captured a moving object if the temperature value is a value other than TN. If the temperature value is TN, a stationary object has been captured. Is shown. Next, a moving object image is created by the moving object image creating means 51 from the difference image 2 between the image A and the image B. Creation of a moving object image is performed as follows. That is, when the temperature difference value of each pixel in the difference image is dT (x, y) and the temperature value of the pixel of the same coordinate of the moving object image is V (x, y), dT (x, y) <0 If V (x, y) = T1 (x, y) dT (x, y)> 0 If V (x, y) = T2 (x, y) dt (x, y) = TN V ( By performing processing such as x, y) = TN for all pixels in the difference image, a moving object image can be created. next,
A frequency distribution H (x, y,
T) is created. The frequency distribution is created as follows. That is, when the pixel value at the coordinates (x, y) in the moving image is V (x, y) and the frequency distribution for the pixels at the same coordinates in the thermal image is H (x, y, T), H (x, y, V (x, y)) ← H (x, y, V (x, y)) + 1 is applied to all pixels in the image of the moving object to obtain each pixel in the image. A frequency distribution can be created for each temperature.

By performing the above-described processing from the detection of the thermal image to the generation of the frequency distribution on all the measured thermal images, the frequency distribution is constructed for each pixel, and if sufficient time elapses, , The temperature of the pixel has a frequency distribution that reflects the temperature range in which it can be determined that a human is caught. However, depending on the accuracy of the two-dimensional infrared sensor 2, it is advisable to provide a frequency threshold value as described below and exclude a temperature value with a small frequency.

Further, the temperature range for cutting out a person from the image A, the frequency distribution, and the appropriate frequency threshold value k contained therein is determined by the clipping temperature determining means 53. Then, the human detection means 42 calculates the temperature value of the pixel at the coordinates (x, y) in the image A as T1 (x, y), and the frequency distribution of that pixel as H (x, y, T). When the temperature value of the pixel at the same coordinate is set to M1 (x, y) by the temperature cutout temperature determining means 53 indicating the threshold value k and the temperature cutout range for each pixel, H (x, y, T1 (x, y)) <k, M1 (x, y) = TN H (x, y, T1 (x, y)) ≧ k, M1 (x, y) = T1 (x, y) By doing for all pixels of
Human pixel blocks can be extracted from image A.
Further, a human pixel block is created by extracting only pixels representing a human from the image B by cutting out the temperature B from the image B, the frequency distribution, and the frequency threshold value and the human detecting unit 42. In this process, the human region is extracted in the same manner as when extracting the human pixel block from the image A.

Further, among the temperature values having the frequency equal to or higher than the frequency threshold value k, the lowest value and the highest value are selected, and if the temperature of the pixel is between the lowest value and the highest value, the pixel catches a human being. There is also a method of determining that This method is effective immediately after the operation of the device, and is effective when the frequency distribution does not sufficiently reflect the temperature range in which it can be determined that the temperature of the pixel catches a human. According to the configuration of the present embodiment as described above, the human can set the temperature range in which it can be determined that the human is captured for each pixel even when there is a large variation between the detection elements of the sensor or when noise is added to the image. It is not necessary to do so, and as the human region detection unit is used, the temperature range in which it can be automatically determined that a human is captured can be set for each pixel.

In this embodiment, two consecutive images are used, but the same processing can be performed when three or more thermal images are used, and more differential images can be obtained. It has the advantage of faster construction.

In the embodiment, the two-dimensionally arranged frequency distribution is used, but for example, by mechanically scanning the detecting means in which the detecting elements are arranged one-dimensionally in the vertical direction in the horizontal direction. As a result, in the case of using the two-dimensional thermal image detection means, it is not necessary to create the frequency distribution for each pixel, but it is sufficient to create it for each row of the image, so that the storage capacity of the frequency distribution storage means is reduced. It is possible to extract the human region from the thermal image according to the common temperature range for each row obtained from the frequency distribution. The same applies to the case of mechanically scanning the detecting means in which the detecting elements are arranged one-dimensionally in the horizontal direction in the vertical direction, and a frequency distribution is created for each column of the thermal image, and for each column obtained from the frequency distribution. Extraction of a human region from a thermal image is performed according to a common temperature range.
Further, when a single detection element is mechanically scanned in the horizontal direction and the vertical direction, resulting in a two-dimensional thermal image detecting means, it is not necessary to create a frequency distribution for each pixel. Needless to say, only one frequency distribution is required, and the human region is extracted from the thermal image according to the temperature range common to all pixels obtained from the frequency distribution.

Next, the configuration of the personal information extraction unit 14 of FIG. 1 will be described with reference to FIG. Personal information extraction unit 14
Then, from the human pixel block obtained from the human area detection unit and its representative point, and the thermal image stored in the image storage unit 12, the image information detection unit determines the feature amount for estimating more detailed information about the human. calculate. Reference numeral 60 is a person feature detection unit, 61 is a posture determination feature amount detection unit, 62 is a position determination feature amount detection unit, 63 is a skin temperature determination feature amount detection unit, and 64 is a clothing amount determination feature amount detection unit.

First, a method of detecting the number of people will be described. Conventionally, when detecting the number of people in an image from an image,
The area of a human part is detected from the image, it is judged from the shape of the area whether it is a human, and the number of people is calculated according to how many there are. However, with this method, it is difficult to detect the number of people well unless the shape of the region is accurately recognized from the image. Also, when humans overlap, it becomes difficult to calculate the number of people. Therefore, in this embodiment, the number of people is calculated by the pixel number calculating means, the block form determining means, and the human pixel block counting means.

FIG. 8 shows the configuration of the person number detecting means 60. In FIG. 8, 70 is a pixel number calculating means, 71 is a block form determining means, and 72 is a human pixel block counting means. The number of humans is determined from the number of human pixel blocks in the room detected by the human area detection unit 13 and the number of pixels in each block. The number of detected pixel blocks is used to determine the number of people. Further, since humans may be detected in an overlapping manner, the number of people is determined by using the shape of each pixel block. In this case, the shape information is determined by the image information detection unit 3
The number of people detecting means using the neural network of is used. FIG. 9 shows a simple method of discriminating the number of people when humans overlap. Assuming that the number of pixels in a pixel block is GA as a simple method for determining the number of persons, the number of persons is one when GA <TH1 and two when TH1≤GA <TH2. N threshold values (TH1, TH2, ...
・ 、 THN) Prepare. However, TH1, TH2, ..., THN are positive integers.

With this configuration, the number of people can be determined by the image input means for inputting a visible image by determining the human region and the number of pixels forming the region based on edges, lines and colors.

Next, in the determination of the number of people from the above pixel blocks, it is necessary to determine the threshold value according to the image input device, its installation position, the image quality, the image resolution and the like. Therefore, in order to simplify such adjustment, there is a method of using a learning function such as a neural network. FIG. 10 shows an embodiment of the neural network.
Reference numeral 73 is a neural network for estimating the number of people. The neural network 73 receives the number of pixels of the human pixel block obtained from the number-of-persons characteristic detection unit 60, the representative point, and the form type, and calculates the number of persons in the output. Various engineering models can be applied to the neural network 73. For example, when a multilayer perceptron is used, back propagation (error inverse propagation method) is famous as a learning method. This learning gives the neural network the input information and the desired output information at that time,
Adjust the internal condition. In the above case, the feature signals 2 and 3, the human pixel block representative point and the number of persons in the room at that time are given to the neural network for learning. This configuration can reduce the work of empirically determining the threshold value in order to adjust the internal state by learning. Further, as an alternative to the neural network, it is also possible to replace it with statistical processing such as regression analysis.

Next, a method of detecting the foot position will be described. FIG. 11 is a diagram showing the configuration of the first embodiment for detecting the foot position. In FIG. 11, 90 is a column data extracting means for extracting column data, and 91 is a foot position estimating means for detecting a foot position. The foot position estimating means 91 includes an algorithm method, a statistical method such as regression analysis, and a neural network method. In this embodiment, a method using a neural network is shown.

The human pixel block detected by the human area detection unit 13, the coordinates of the representative point of the pixel block, and the peripheral pixel values of the human pixel block on the thermal image are input to the position determination feature amount detection unit 62. First the human pixel block
It is input to the column data extraction means 90. The column data extracting means 90 inputs the data of one vertical column including the lowermost pixel of the human area to the foot position estimating means 91. When there are a plurality of pixels at the bottom edge, there are a method of extracting a column including a pixel having the highest pixel value, and a method of extracting by a center or left and right positions. When the resolution of the thermal image is 8 pixels in the vertical direction, 8 temperature values represented by each pixel are extracted. Based on the column data extracted by the column data extraction unit 90, the foot position estimation unit 91 using a neural network calculates the human foot position. It is assumed that the foot position estimating means 91 using a neural network is in a state where it is considered that the error has converged by performing learning by previously giving a sufficient number of data and an appropriate teacher signal by using the back propagation method. . In addition, in the foot position estimating means 91, in addition to the multilayer perceptron,
Other neural network models such as LVQ can also be used.

FIG. 12 is a schematic diagram showing a correspondence relationship between a standing human thermal image and a human body. a to o represent each pixel of the thermal image, and the hatched portion represents a region corresponding to the human body. In this case, since most of the pixels of f, g, and h occupy most of the human body part, the clipping is performed well, but in the case of i, the position of the foot is near the center of the pixel, so the upper half is temperature. Is high and the lower half is cold. Since the area of the human body portion occupying one pixel is small, the pixel value at this time overlaps the temperature of the human portion and the temperature of the background portion, resulting in an intermediate value, which falls below the cutout temperature range and is actually cut out. Only three pixels of f, g and h are displayed. Even if the foot position is estimated from this, it cannot be accurately obtained, but by inputting the value of i into the neural network,
It is possible to obtain an accurate foot position.

When an algorithm is used as a method other than the neural network, the foot position estimating means 9
1, using only the values of the three pixels h, i, and j, h
Is the temperature of the human being, and j is the temperature of the background, where α is the proportion of the human being in the pixel i, α is obtained from i = α * h + (1-α) * j The position can also be calculated.

FIG. 13 illustrates the feature quantity in the position determination feature quantity detection unit 62. External information including people 11
0 is taken as a thermal image 112 by the infrared sensor 111, and the human block in the shaded portion is cut out by the human area detection. The center of gravity of the detected temperature value (pixel value) of the person's block is obtained, and the value of the pixel A at the lowest point of the person's block obtained by lowering the center of gravity in the vertical direction is taken as the pixel value (A). The value of the pixel B that is one pixel higher in the vertical direction with respect to this pixel is the pixel value (B), the value of the pixel C that is one pixel lower in the vertical direction is the pixel value (C), and the value of the pixel D next to it is the pixel value. Value (D)
And Also, the background temperature of the person's block or the average value of several horizontal pixels of the pixel C is calculated as the foot temperature, the average temperature of the entire person's block, and the reference temperature inside the sensor as the feature amount. These feature quantities are determined by the image information detection unit 3
The position 114 of the person is estimated by inputting to the internal position estimation neural network 113.

Next, a method of detecting the posture will be described.
In posture detection, there are three types of postures to be detected: standing posture, lying posture, and sitting posture. FIG. 37 is a diagram showing the configuration of the first embodiment for detecting the posture. In FIG. 37, 120 is a human pixel block obtained from the human area detection unit 13 and a rectangular block including a human is extracted from the thermal image, and side pixel number detection means A and 121 for detecting the vertical size of the rectangular block are A side pixel number detecting means B for detecting the lateral size of the rectangular block, 122 is a converting means for standardizing the rectangular block into a block having a size of 3 × 3, and 123 is an attitude estimating means. Posture estimation means 123
There are various methods such as statistical processing and a method using a neural network, but in this embodiment, a multilayer perceptron that has been preliminarily learned by the back propagation method was used.

Regarding the human pixel block detected by the human area detecting unit 13, the side pixel detecting means A and B 120 and 121 extract a rectangular block including the periphery from the human pixel block and the thermal image, The number of horizontal pixels is detected and the number of vertical side pixels and the number of horizontal side pixels are output, respectively. Conversion means 1
22, 3 human pixel blocks of various sizes
It is standardized into a signal corresponding to a block of × 3 pixels and is output as a fixed pixel signal. By inputting the number of pixels on the vertical side, the number of pixels on the horizontal side, a fixed pixel signal, and the foot position information obtained from the foot position detecting means 17 as necessary, the posture estimating means 123 using a neural network calculates Human posture is obtained. The posture estimation means 123 using a neural network performs learning by previously using the back propagation method and giving a sufficient number of data of different postures at various positions and an appropriate teacher signal to converge the error. Shall be in a state considered to be. It should be noted that the posture estimation unit 123 can use other neural network models such as LVQ other than the multilayer perceptron.

FIG. 15 shows an example of processing for detecting the posture. Reference numeral 130 is a measured thermal image, and 131 is a human pixel block obtained from the human area detection unit 13. This human pixel block is converted into a rectangular block 132 by detecting the vertical and horizontal lengths using a side image detecting means A120 and a side pixel detecting means B121 using a thermal image. Rectangular block 1
It is conceivable that 32 has various sizes because the size of the human area is not constant depending on the position of the person. This cannot be directly input to the multilayer perceptron in which the number of neurons is fixed, so data must be standardized by some method. Here, 122
A case will be described below in which the conversion means standardizes the data into 9 × 3 × 3 pixel data using the area ratio. In Figure 16,
The procedure for normalizing a 4x5 rectangular block is shown. 4x
If the respective values of 20 pixels of 5 are a to t, and the respective values of standardized 3 × 3 9 pixels are A to I, they are standardized using the area ratio, for example, the value of A is A = { 0.25 * 0.2 * a + 0.25 * (0.33-0.2) * e + (0.33-0.25) * 0.2 * b
+ (0.33-0.25) * (0.33-0.2) * f} / (0.33 * 0.33). Similarly, B to I can be obtained.

In addition, by inputting auxiliary data together with foot position information and vertical and horizontal pixel numbers into a 12-input 4-output multi-layer perceptron that has been learned in advance by the backpropagation method, the posture of a human in the room can be obtained. To be

When learning with a neural network,
It is also possible to output the data including the orientation in each posture by adding the data obtained by changing the orientation of the human body from the two-dimensional infrared sensor to the learning data. As for the target posture, it is possible to handle the posture in consideration of the orientation and the subdivided types of posture.

Next, the feature amount of the posture determination feature amount detecting section 61 will be described with reference to FIG. FIG. 17 shows the flow of posture determination from a thermal image to posture determination according to the present invention. In FIG. 17, the area of the person in the thermal image 140 is indicated by diagonal lines. When the above-mentioned person's area is detected, a pixel area in a square (thick solid line portion) interpolating the area is cut out, and the feature extraction unit 141 performs area determination for 3 × 3 nine posture determinations. There are various methods for calculating the nine (3 × 3) feature amounts. For example, if the square of the thermal image is 2 × 2 as shown in FIG. 17, each pixel is G (1),
If G (2), G (3), and G (4), then nine feature quantities, F (1),
F (2), ..., F (9) Becomes In addition, as other features, the position of the center of gravity of the human block or the position of the pixel at the lowest point of the human block in the thermal image in the vertical direction, the number of blocks of the human block, and the number of blocks of the human block Calculate the average temperature. The neural network 1 in the image information detection unit 3 uses these feature quantities.
The value is input to 43 and the value 144 indicating the attitude as a result of the attitude determination is output. The outputs 145 and 146 of the neural network in FIG. 17 are used as selection signals or correction signals for position determination described later.

Next, another configuration of the image information detecting section 3 will be described with reference to FIG. It is assumed that the image information detection unit 3 has a structure of a foot position detection unit 17 having a posture detection unit 18 and a position determination unit for each posture. The position discriminating unit may have a single configuration when the posture is not classified, or the posture classification and other posture classifications, for example, when the sitting position is finely classified into a chair sitting position and a flat sitting position, only the posture classifications can be increased. In the embodiment of the present invention, a processing flow is shown based on the configuration of FIG.

The feature amount 150 for posture determination calculated by the image processing unit 3 is input to the neural network configured as a posture determination unit as shown in FIG. 17, and the posture 144 is output. The neural network of the posture determination unit actually learns a large number of people with different clothing and body shapes in the scanning area of the thermal image by detecting the feature amount data at multiple positions and learning the posture at that time as a teacher signal to the neural network. I do. The learning method differs depending on the type of neural network, but as a well-known method, there is learning using the steepest descent method called back propagation that adjusts the internal state to reduce the square error between the output and the correct answer. . The signal 152 and the signal 153 from the attitude determination unit will be described in the overall flow after the description of the position determination unit. In the position determination units 161, 162, 161 is for sitting and 162 is for standing. Each estimation unit is composed of a neural network as shown in FIG. 13, and the estimated human position 154,
155 is output. The neural network of each position determination unit actually detects a large number of people who have different clothing and body shapes in the scanning area of the thermal image and detects the feature amount data at a large number of positions, and uses the positions from the sensors at that time as teacher signals. Learn to the neural network. Although the learning method differs depending on the type of neural network as explained in the posture determination unit, for example, it is a well-known method that adjusts the internal state so as to reduce the square error between the output and the correct answer, which is called back propagation. There is learning using the steep descent method. The feature quantity 151 for position estimation is the position determination units 161, 1
The position determination results 154 and 155 are output to the input terminal 62. It is necessary to select or process these two outputs and output them. There are the following two methods for roughly selecting 154 and 155. (1) Select 154 or 155 based on the output result 144 of the posture determination unit. (2) Output values 152, 153 and 154, 1 of the posture determination unit
Process 55 and output. The method (1) is generally performed based on the IF-THEN rule. In other words, the simplest method is that the IF output 144 outputs the sitting THEN 154 and the IF output 144 outputs the standing THEN 155. However, in the method (1), even if the posture has an ambiguous value (when it is impossible to determine whether the posture is sitting or standing), either of them must be forcibly selected.

Therefore, as shown in FIG. 18, the output D152 for processing X154 and X155 from the posture determination unit,
D153 is output, the following processing is performed, and the position estimation value 9 of the image information detection unit is output.

Y = D152 × X153 + D153 × X154 This advantage is that X152 and X153 output 1 or 0 if the posture determination result is clear, but 0 to 1 if ambiguous.
The position estimation value 163 can estimate a more accurate position because a value corresponding to the ambiguity up to is output. The adjustments of the outputs 152 and 153 of the posture determination unit 160 are as follows.
Position determination units 161, 1 for learning the neural network
This can be performed by learning the sitting and standing learning data after learning of sitting and standing by 62. These learning methods are, for example, the above-mentioned back propagation method as a structured neural network learning method. Although the personal information extraction unit 14 is included in the image processing unit 2 in the present invention, it is naturally possible to incorporate the personal information extraction unit 14 into the image information detection unit 3 in the subsequent stage.

Next, the personal information extraction section 14 will be described in detail. The personal information extraction unit 14 detects the number of people and the position of the feet, detects the posture of the person in the room, and also detects the skin temperature and the amount of clothing in the first embodiment. FIG. 19 shows the first
It is a block diagram of the personal information extraction part 14 in the Example.

Next, the skin temperature detecting method will be described. The skin temperature detecting means 20 detects the skin surface temperature of a person who is in the room. FIG. 20 shows the skin temperature detecting means 2.
It is a block diagram of 0. The skin temperature detecting means 20 will be described below. Reference numeral 360 denotes a block dividing unit that divides the human pixel block based on the human pixel block obtained from the human region detecting unit 13 and the posture obtained from the posture detecting unit 18 according to the body part, and 361 denotes the block divided by the block dividing unit 360. A block temperature calculating means for calculating the average temperature of the block 362 compares the respective temperatures obtained by the block temperature calculating means 361, and determines the block corresponding to the skin temperature at which the body surface appears and the block covered with clothes. It is a temperature comparison means for separating.

The operation of the skin temperature detecting means 20 configured as described above will be described below. The human pixel block obtained from the human area detection unit 13 is divided into a maximum of six parts using the posture information so that the human body includes the head, chest, left arm, right arm, waist, and legs. In each block division, a human pixel block division standard pattern is determined in advance for each posture and pattern recognition is performed. At this time, since the size of the human pixel block changes depending on the position of the person, the block division is performed using the ratio used for the standardization of the conversion unit 122 of the posture detection unit 13. 21 (a) and 21.
An example of block division is shown in (b). FIG. 21A shows a human pixel block of a standing person in a room. The size is standard for simplicity. In this case, A is a block containing the head,
B is a block including the chest, C is a block including the right arm, D
Is a block including the left arm, E is a block including the waist, and F is a block including the legs. The division into the above six blocks is because each block corresponds to a part of the body where exposure and coating may occur independently. Figure 21
(B) is a human pixel block of an occupant sitting in a chair facing left. In this case, since the right arm is hidden, there are five divisions. Next, the block temperature calculation means 3
At 61, the average pixel value of each divided block is calculated. When the body surface is exposed, the average block temperature is high, and when it is covered with clothes, it is low.
Therefore, the temperature comparison means 362 then compares the predetermined threshold temperature with the block average temperature, and outputs the block average temperature having the maximum average value among the blocks having a temperature higher than the threshold temperature. . In other words, if the exposed part of the body surface is large like the face, the average block temperature will be high, but if the part covered by the background or body clothes is in the same pixel, the average block temperature will be low. Therefore, it can be judged that the block having the highest average temperature is the largest body part of the exposed portion. The skin temperature is detected with the above configuration.

Next, a method of detecting the clothing amount will be described. The clothing amount detecting means 19 detects the clo value of the person extracted from the thermal image. FIG. 22 is a block diagram of the clothing amount detecting means.

In FIG. 22, 363 is an area comparison means.
The above is the same as the configuration of the skin temperature detecting means in FIG. Here, the clothing amount to be detected is called a clo value, and for example, it is 0 in a naked state, 0.1 in pants only, 0.3 in upper and lower underwear, and 0.5 in short-sleeved and trousers summer clothes. The value is 0.7 for light work clothes with long sleeves and trousers, 1.0 for indoor clothes in winter, 1.5 for three-piece suits, and 3.0 for overcoats in winter. ing. The amount of clothing is determined by how much of the surface area of the body is covered with clothes and how thick the covered portion is. So, regarding the thickness of clothes,
In the temperature comparing means 362 in the skin temperature detecting means 331,
By providing a temperature threshold value, it is possible to separate the clothing block whose body surface is covered with clothes from the exposed body surface block, and it is estimated by taking the temperature difference. That is, when the temperature difference is large, it can be determined that the amount of clothing is thick, and when the temperature difference is small, the amount of clothing is thin. On the other hand, regarding the extent to which the body surface area is covered with clothes or the like, the area comparison unit 363 counts the number of pixels of the blocks separated into the clothing block and the body surface block and compares the area ratios. This allows the ratio of the clothing surface area to the naked body surface area to be estimated from the area ratio. Further, in the area comparing means 363, c is calculated from the thickness of the clothing and the ratio of the clothing surface area to the naked body surface area.
The lo value is associated and output. Although the clo value is output here for the purpose of calculating the comfort level, these values can be directly output when the comfort level calculation formula using the coverage ratio of the body surface and the clothing surface temperature is used. Further, it is also possible to previously learn the relationship between the feature amount and the clothing amount for detecting the clothing amount in the neural network and obtain the clo value using the neural network.

Next, FIG. 23 shows the configuration of the environment information extracting section 15. The environment information extraction unit 15 is not necessarily an essential component of the image processing apparatus, but is a component added as necessary. 370 is a foot temperature detecting means for detecting the temperature near the foot from the foot position information and the thermal image, 3
Reference numeral 71 is an indoor part temperature detecting means for calculating an average temperature for each part of the wall and floor from the thermal image, and 372 is a foot temperature detecting means 37.
Foot temperature and room temperature detecting means 3 detected by 0
Radiation temperature detecting means for detecting the radiant temperature of each person in the room from the temperature of each part in the room detected from 71 and the foot position information output from the foot position detecting means 17. less than,
Each means will be described. The foot temperature detecting means 370 will be described. This means is intended to easily and accurately detect the floor surface temperature near a person in the room. When trying to detect the floor surface temperature near humans based on the indoor thermal image, the problem is that it is difficult to identify where the floor surface is and that the temperature of the floor surface is not always uniform. Since it does not exist, the temperature of the floor surface near humans must be extracted. In the present invention, a method is used in which a person inside a room is first cut out, the foot position thereof is detected, and then the temperature of the floor surface, which is considered to have a great influence on the person, is extracted from the pixels around it.

FIG. 24 is a block diagram showing the structure of the foot temperature detecting means 370. In FIG. 24, reference numeral 380 is a peripheral pixel extraction means for extracting pixels around a person's foot, and 381 is a foot temperature calculation means for detecting a foot temperature. The foot temperature calculation means 381 can be configured by an algorithm, a method using statistical processing, or a method using a neural network. In this embodiment, a neural network is taken as an example.

First, the human pixel block in the room and the signal indicating the representative point of the pixel block and the thermal image are input to the peripheral pixel extracting means 380. The peripheral pixel extraction means 380 extracts the values of the four pixels that are considered to be closest to the human foot region from the thermal image and uses them as peripheral pixel signals. Information about the surroundings of human feet, such as peripheral pixel signals and foot position information,
It is input to the foot temperature calculating means 381 which is a multi-layer perceptron. The foot temperature calculating means 381 using a neural network uses a back propagation method to obtain a sufficient number of data and an appropriate teacher signal in advance, using the input information and the teacher information for the input information as the radiation temperature actually measured at that position. It is assumed that learning is performed by giving the error and that the error is considered to be converged. It should be noted that, in this foot temperature calculation means 381, it is possible to use a neural network model such as LVQ other than the multilayer perceptron. In this way, in the foot temperature detecting means 370, processing is performed based on the peripheral pixel signal,
The floor temperature at the feet of humans in the room can be obtained.

FIG. 25 shows an example of processing in the foot temperature detecting means 370. Reference numeral 390 is thermal image data captured in the image storage unit 12. Reference numeral 391 is a human pixel block obtained from the human area detection unit 13. Next, based on the human pixel block 391, the peripheral pixel extraction means 380 detects the pixels corresponding to the foot position of the human and extracts the pixels around the foot of the human. This time, the foot peripheral pixels are set to 4 pixels. This is based on the detected foot position, and is basically two pixels outside the human area outside of the detected foot position. However,
If the human region is located at the edge of the thermal image data or if there is another human region in close proximity, effective pixels cannot be obtained. In that case, the foot temperature may be detected from a small number of pixels, but when a multi-layer perceptron is used for the foot temperature detecting device, it is desirable that the number of input data is uniform, and therefore, as shown in 394 of the surrounding pixels. In addition, a method is used in which a priority list is created centering on the pixels corresponding to the foot position and four pixels are extracted in descending order of priority. The values of the four pixels extracted in this way, the foot position information, and the like become peripheral pixel signals.

There is also a method of using a foot temperature detecting means for averaging four pixel values obtained from the peripheral pixel extracting means 380 to obtain a foot temperature instead of the foot temperature detecting means 381 using a neural network. It can be realized by simple processing.

Next, the indoor part temperature detecting means 371 will be described. If information about whether the mounting position of the two-dimensional infrared sensor 11 on the wall is at the center, right, or left, and the size of the room, the boundary between the wall and the floor appears at the position on the thermal image. Can be calculated in advance. Therefore,
The indoor part temperature detecting means 371 obtains information about the size of the room and the installation position of the air conditioner by attaching a switch to the air conditioner, and calculates the pixels of the boundary line between the wall and the floor based on the information. , The areas surrounded by the respective boundary pixels are a wall and a floor in three directions. Further, the average temperature is calculated for each area, and the average temperature for each site in the room is calculated. Here, the temperature may be partially different even on the same floor or wall due to the influence of the heat-generating object, the condition of the adjacent room, the solar radiation, etc. Therefore, each part may be divided into a plurality of blocks to be handled. FIG. 26A is an example of how the thermal images of the wall and the floor are viewed when the air conditioner is centrally installed on one wall surface of an 8-mat square room. Here, the floor surface is divided into six. 390 is a left wall, 391 is a back wall, 392 is a right wall, 39
Floor blocks 3 to 398 are divided into six.
FIG. 26B is an example of how the thermal images of the wall and the floor are viewed in the case of installation on the left. 400 is a left wall, 401 is a back wall, 402 is a right wall, and 403 to 408 are floor blocks divided into six. In this way, by obtaining information about the size of the room and the installation position of the air conditioner, it is possible to know in advance which position on the thermal image each part in the room will appear.

Next, the radiation temperature detecting means 372 will be described. The radiation temperature for each individual is calculated from the temperatures from the respective parts of the wall and floor in the three directions and the temperatures near the feet. Conventionally, as a device that considers radiation, there is an air conditioner that measures the temperature of multiple areas separated on the floor surface and preferentially warms the cooler floor surface area.However, the radiation temperature is a device control index. When used as a part of, the floor alone is not enough, and radiation from walls and windows is often large when it is cold outside.
In addition, the floor and walls have different physical properties and wall structures, and the direction of radiation to the human body differs between the floor and each wall, so the effects from each part such as the wall and floor should be considered separately. It is more accurate to calculate radiation. Figure 2
7 shows the configuration of the radiation temperature detecting means 372. 410 is
An average temperature calculating means for calculating the average temperature of the pixel blocks of each indoor part obtained by the indoor part temperature detecting means 371;
Reference numeral 411 is a radiation temperature calculation means for calculating the radiation temperature for each individual from the foot temperature obtained by the foot temperature detection means 370 and the average temperature for each part obtained by the average temperature calculation means 410. In the radiation temperature calculation part, the foot temperature near the occupant,
Radiation from each part is calculated for each person in the room by multiplying the average temperature of the floor block and the average temperature of the walls in the three directions in the room by a coefficient such as a view coefficient whose total is 1. For the floor block, it is possible to use a part of the floor surface divided according to the position of the person in the room.

Next, a different configuration of the environment information extraction unit 304 will be described. In FIG. 28, reference numeral 420 is an auxiliary heating detecting means, 421 is a floor heating detecting means, and the above is the same as the configuration of the environment information extracting unit 304 in the first embodiment. Here, the auxiliary heating is a combustion or electric heating type heater such as a stove or an oil fan heater.
In addition to normal floor heating, floor heating includes electric carpet. Since the auxiliary heating is higher than the human body temperature, the auxiliary heating detection means 420 detects it by providing a certain temperature threshold value higher than the human body in the measured thermal image.

Further, since the floor heating has a temperature close to the human body temperature, it is once detected as a human pixel block by the human area detecting unit 13. Since the area is larger than the human body and the shape is usually a quadrangle, the size and shape of the cut out human pixel block are set to a range to determine whether floor heating is present. In addition, since the floor heating does not change much with time except when the power is turned on and off when the floor heating is detected, it is possible to detect the change in the human pixel block by examining the change in the cut out human pixel block for a predetermined period. It is also possible to detect it on the condition that it does not exist. As a result, it is possible to correct the influence of heat transfer directly from the floor surface due to the floor heating and the influence on the calculated comfort degree due to the influence of high radiation due to combustion of the stove. Further, for example, when there is floor heating or auxiliary heating, it is possible to control the air conditioner while keeping the comfort level low for the purpose of energy saving. Next, a different configuration of the image processing unit 2 will be described. Since the configuration is different thereafter, the image processing unit 430 is used. The image processing unit 430 will be described with reference to FIG. Four
Reference numeral 31 is a human area correction unit, 432 is an accumulation processing unit, and 433 is an indoor part determination unit.

First, the human area correction section 431 will be described. In the human area detection unit 13, for each pixel of the image showing the temperature distribution, if the temperature value of the pixel is within the temperature range in which it can be determined that the pixel captures a human, the pixel is validated. The human region image is created by performing processing such as leaving the pixels as valid pixels and removing them as invalid pixels otherwise. However, the temperature value of a pixel that originally captures a human does not match the temperature range in which it can be determined that a human is captured due to factors such as human clothes, and that pixel is processed as an invalid pixel. There may be a case where a human region image is created in which a human region that should be one region is divided into a plurality of regions. The human area correction unit 431 compensates for the above-described problems of the human area detection unit 13.

An embodiment of the human area correction unit 431 will be described with reference to the drawings. In FIG. 30, 4
Reference numeral 40 is a cutout pixel connecting means, 441 is an isolated point searching means, and 442 is a peripheral pixel searching means.

The human pixel block detected by the human area detecting unit 13 is cut out together with the representative points, and the cut-out pixel connecting means 440 is used.
Entered in. The cut-out pixel connection unit 440 generates information as to whether or not there is a cut-out pixel in the eight pixels around the two-dimensional coordinates of each pixel of the human pixel block. This connection information is input to the isolated point search means 441. Based on the connection information, the isolated point search means 441 confirms whether or not there is an isolated point in which the pixels cut out into eight surrounding pixels do not exist, and outputs isolated point information. The isolated point information is input to the peripheral pixel search means 442, and the isolated point information 2
It is searched whether or not there is a clipped pixel in the surrounding 16 pixels, which is one pixel away from the dimensional coordinates. If there is a clipped pixel in the surrounding 16 pixels, it is connected to the clipped pixel. If it does not exist, a process of erasing the corresponding isolated point is performed. As a result, the corrected human pixel block is output from the peripheral pixel search means 442, and the isolated pixel is not included in it. However, when there is a possibility that an isolated point exists depending on the position of a pixel on the two-dimensional coordinates of the image, an exceptional process that does not perform erasure and connection is inserted depending on the coordinates of the isolated point. Finally, the representative point is calculated based on the corrected human pixel block again.

The human area correction unit 431 described above should originally be one human area due to the noise removal on the image, the determination of other heating appliances as a small human area, and the appearance of human clothes. Even if something is divided, by deleting a small human area as an isolated point or by complementing between divided human areas,
It is possible to correctly extract a human area and detect human information from the human area image.

Next, the accumulation processing section 432 will be described. If the shape of the area cannot be accurately recognized from the image, the accumulating unit 432 can detect information on the human and the room, but it can effectively use the information by associating the information on the human and the room with the motion of the human. And easily calculate human and indoor information.

FIG. 31 shows a block diagram of the accumulator. Figure 3
In 1 the reference numeral 450 designates a cumulative storage means A, 45 for human positions.
Reference numeral 1 is a stationary object determining means, 452 is a stationary object pixel position storing means, 453 is a cumulative storing means B, and 454 is an indoor representative point determining means.

The human pixel block representative points obtained by the human area correction unit 431 are stored in the accumulating storage means A450 having the same resolution as the image. This cumulative storage means A45
In 0, the number of times of the pixel position of the human representative point detected from the M thermal images is stored. The cumulative position information from the cumulative storage unit A450 where the cumulative storage of the M thermal images is completed is performed by the stationary object determination unit 451 to distinguish between a person and a stationary object, and the detected pixel position of the stationary object is stored. Stored in 452. The cumulative storage means A450 processes the data and stores it in the cumulative storage means B453. In the accumulative storage means A450, 1 is stored in the place where the human is and 0 is stored in the place where the human is not. Cumulative storage means A450
Is erased when the data is stored in the cumulative storage means B453, and the cumulative storage is performed again. Cumulative storage means B453
When the data from the cumulative storage means A450 is written N times, the indoor representative point determination means 454 performs the processing described below and erases the data. After the cumulative storage unit B453 is also deleted, the same processing is executed from the beginning. By processing the data of the accumulative storage means B453, the installation position of the two-dimensional infrared sensor in the room, the position of the wall surface or floor surface in the room, the volume, and the like are calculated.

Next, detailed processing of the stationary object determining means 451 shown in FIG. 31 will be described. The cumulative storage A450 stores the number of times each pixel of the thermal image is detected as a human position. For example, the result at the coordinate (X2, Y2) on the two-dimensional cumulative storage means A450 is Sx2, y2. This number of times Sx2, y2 is compared with a threshold value TH1 (a positive integer), and TH1
A static object list (stored) is created as a static object having a larger value. In FIG. 32, M is 120
Since TH2 is shown as an example, TH2 is set to 100. As described above, among M thermal images, the position of an object that stores the number of times of cutting at the same position and has a value close to M in the cumulative memory is a stationary object (for example, a stove) accidentally cut out with a person. And electric carpets, etc.) can be determined. That is, since a stationary object does not move, a numerical value close to the M value is stored in the stored position, so that it can be distinguished from a human body having some movement.

Next, FIG. 33 shows a process of storing the information in the cumulative storage means A450 in the cumulative storage means B453. The value of Sx2y2 in the cumulative storage means 1 is compared with the threshold value TH2, and if Sx2y2> TH2, 1 is written in the coordinate (x2, y2) on the cumulative storage means B453, and nothing is done otherwise. By repeating this process N times, when the area 1 of the cumulative storage unit B453 is confirmed, the range of movement of the person is indicated. The comparison with the threshold value TH2 is performed in order to remove the result of the processing that is erroneously detected as a person. The information in the cumulative storage means B453 is processed by the indoor representative point determination means 454,
For example, the processing as shown in FIG. 34 can be performed. The indoor representative point determining means 454 will be described below. Since 1 is stored in the position where the person is present and 0 is stored in the place where the person is not stored in the cumulative storage means B453, it is divided into three in the horizontal direction.
A cumulative total of 1 is calculated for each of the divided areas of the blocks A, B, and C. The total of the obtained blocks is SA, SB, S
Determine the value of C as follows to determine whether the air conditioner is installed on the wall in the center, left, or right.

If SA <TH and SC ≥ TH, install on the C side. If SC <TH and SA ≥ TH, install on the A side. In other cases, it will be installed centrally.

Further, the indoor representative point determining means 454 also executes each part such as the floor and the wall surface in the room. FIG. 35 shows an example of determination of each part in the room. In FIG. 35, cumulative storage B45
The movement range of the human obtained from 3 is the range surrounded by A, B, C, and D. The coordinates of A are (X1, Y1) and the coordinates of B are (X2, Y1).
2), if the coordinates of C are (X3, Y3) and the coordinates of D are (X4, Y4),
In A, it is assumed that X1 is the threshold value TX or more and Y1 is the largest point. In B, it is assumed that X2 is less than the threshold value TX and Y2 is the largest point. In C, X1 is the threshold value TX or more and Y1 is the smallest point. In D, X2 is less than the threshold value TX and Y2 is the smallest point. Under the accumulation of a sufficiently long time, these A, B, C, and D become representative points that represent the form of the floor surface, which is the range of movement of a person in the room. Therefore, the coordinates of each representative point are output. In addition, it is possible to determine a point representative of the floor surface shape even in a room that is not a quadrangle.

The accumulator 432 also detects the position of a person by using an edge, a line, and a color even by an image input means for inputting a visible image, and thus the accumulator 432 of the present invention obtains information about the person and the room. can do.

Next, the indoor part determination section 433 will be described. When the points A, B, C, and D are obtained by the accumulating unit 432 as shown in FIG.
The area surrounded by CD is the floor surface, the area surrounded by ADD-A 'is the left wall surface, the area surrounded by ABB-A' is the front wall surface, and B- The area surrounded by CC′-B ′ is determined as the right wall surface. As for the correspondence between each floor and wall and pixels, prepare an array data structure with the same number and shape as the pixels of the thermal image, and substitute different symbols for each region in the array You can know if it corresponds.

The second embodiment of the present invention will be described below. FIG. 36 is a block diagram of the second embodiment of the present invention. FIG. 36 shows a console unit 460 provided in FIG. 1 for the user to input various information. Console section 4
The processing other than the information from 60 is omitted because it has been described in FIG. The console unit 460 corrects the information that cannot be detected from the thermal image based on the information input by the user, so that the output values of the foot position detection and the posture detection in the image information detection unit illustrated in FIG. , Will be more accurate. As an example, the information from the console section is shown below.

When the infrared sensor is installed at a height overlooking the room and the heat distribution in the room is scanned as a thermal image, the positions of the left and right walls and the front wall surface of the room when viewed from the sensor are determined from the thermal image. Is difficult Therefore, when the user inputs the distance between the front wall surface and the left and right wall surfaces from the sensor, if the position determination result from the position determination unit estimates the position farther than the front wall surface, It can be corrected to the position or treated as a heat-generating object other than a person on the wall. Further, since the temperature distribution of each wall surface can be identified, the comfort level of the thermal environment at the position of the person estimated by the position determination unit can be estimated.

The third embodiment of the present invention will be described below. FIG. 37 is a block diagram of the third embodiment of the present invention. FIG. 37 shows a visible image sensor unit 470 (camera) in FIG.
A visible image processing unit 472 and a visible image information detection unit 47
3 is added to enhance the function of the image processing apparatus.

There is a monitoring system as a method of using an image processing apparatus using a visible camera. 2. Description of the Related Art Conventionally, in a monitoring system, a person constantly monitors a visible image. The technique for recognizing a person from a visible image is being developed, but few have been put to practical use. However, many techniques for detecting a moving object from a visible image have been put into practical use. This is because the S / N ratio of the visible image is improved, and thus the moving object can be easily detected by taking the difference between the temporally continuous visible images. However, it is difficult to accurately capture a person from a moving object because the objects captured by the movement include people other than the person. In addition, it is impossible to detect a person who moves little. On the other hand, an infrared image is very easy to catch a human being as a heat generating object. However, this classification is difficult because heat-generating objects other than humans are also detected. As described above, the apparatus shown in FIG. 37 attempts to detect more accurate human or environmental information by combining detection that is good at visible image processing and detection that is good at infrared image processing.

The flow of processing will be described with reference to FIG. Of the information 5 obtained from the outside world 4, a visible light is detected by the camera 470 to obtain a visible image 473. The visible image 473 is transferred to the visible image processing unit 471 and subjected to preprocessing, temporal continuous image difference processing, and the like. Next, the visible image information detection unit 47
At 2, motion information is detected. This information 475, 4
As 76, the presence / absence of a moving object, the number thereof, the position of the moving object, etc. are transmitted to the image processing unit 2 and the image information detecting unit 3.
Further, based on the information 477, the image processing unit 2 performs area detection with reference to the number of areas and their positions in the human area detection processing. In addition, the visible image information detecting unit 472 associates the presence / absence of moving objects, the number of moving objects, the position of the moving object, and the position of the person detected by the image information detecting unit 3 with the external environment 4
To detect the person information 474 in.

Visible image processing unit 472 and image information detection unit 3
An example of an object to be detected by the mutual information of is shown below. Example 1: Whether the moving object detected by the visible image processing unit 471 or the visible image information detection unit 472 is a person,
The judgment is performed by matching the position of the person detected by the image information detecting unit 3 which detects information from the thermal image. Example 2: A detection mark of a person is shown at the position of the visible image calculated from the position of the person detected by the image information detection unit 3 so that the user can easily check the visible image. Example 3: Only when the posture of the person detected by the image information detection unit 3 is the sitting position, there is a high possibility that the target person is a stationary person, so the position of the person is marked on the visible image. Example 4: When the posture of the person detected by the image information detection unit 3 is sitting, the image information detection unit 3 notifies the information 476 and the visible image processing unit 471 processes the result together with the visible image information detection unit 472. , Person information 474 is created. Example 5: The position and posture of a person detected by the image information detection unit 3 are transmitted to the information 476, the information 476 is used as a teacher signal, and the visible image processing unit is configured by a processing system having a learning mechanism such as a neural network, Evolution of human recognition processing. Example 6: From the motion information signal detected by the visible image processing unit 471 and the visible image information detection unit 472, the detected position of the person is transmitted as information 475, and the information 475 is used as a teacher signal. Learn and evolve the capabilities of processing systems that have learning mechanisms such as neural networks.

Next, FIG. 38 will be described. FIG. 38 shows
The monitoring information detection unit 480 that performs integrated processing by using the information 474 of FIG. 7 and the output of the image information detection unit described above is provided, and the suspicious person warning 481 and the abnormality information 482 are output.
Examples of the information integration include the camera 470 and the sensor unit 1.
If there is movement about the person who entered when the area measured by the
4 outputs motion information, and the image information detection unit 3
Although the position information of the person is output from the suspicious person warning signal 481 if the position of the person does not change after the posture is detected as sitting in the area and the state continues for a certain time or longer. And the abnormality information 482 is output. In this way, information can be used for detection by using temporally continuous information obtained from a visible image or an infrared image and a priori knowledge.

It is also possible to set an area where people are prohibited from entering within the measurement area of the sensor section from the console section 460 to detect a person who enters the area.

In addition to the monitoring system, the present image processing apparatus includes a visible image information detecting section 472 as shown in FIG.
Information is integrated from the output of the image information detection unit 3 and the output of the adaptive learning unit 490 shown in the above-mentioned example, and the personal identification unit 4
It is also possible to identify or track individuals within the detection area at 94. That is, for example, the visible image information detection unit 4
When the information 91 about the color of the clothing of the person or the information about the physique is obtained from 91, and the posture and the foot position information 492 is obtained from the image information detecting unit 3, the adaptive learning unit 492 causes the position and posture of the human being, the physique and the clothing. Learn the features to identify an individual from the colors of. The individual identification unit 494 performs pattern recognition based on the feature amount obtained from the adaptive learning unit 490 to identify an individual. This allows highly accurate personal identification.

Finally, a configuration example of a system in which the above-described image processing device of the present invention is integrated will be shown. The overall configuration will be described with reference to FIG. FIG. 40 is composed of the above-described image processing apparatuses 500 to 502 of the present invention and an integrated processing unit 503 that collectively processes information from the three image processing apparatuses. The three image processing devices 500 to 502 have different angles or different measurement areas. The integrated processing unit 503 temporally and spatially processes the information 504 to 506 obtained from the three devices, and outputs information 507 such as a person's tracking, a person's position and posture, or environmental information 508. Two specific examples will be shown below.

Example 1: The image processing apparatuses 500 to 502 measure and process almost the same area from different points. The observation points and the positional relationship are input from the consoles of the image processing apparatuses 500 to 502. In the image processing apparatus 500, when the person is overlapped and the position and the posture are determined as one person, the image processing apparatuses 501 and 502.
When the information obtained from the information indicates the positions and postures of a plurality of people, the integrated processing unit 503 causes the image processing devices 500 to 50
Accurate information 507 about a person is detected from the arrangement of 2 and output.

Example 2: The image processing apparatuses 500 to 502 measure and process areas in different directions. Image processing device 5
Each observation point and arrangement relation are input from each console section of 00 to 502. A person detected by the image processing apparatus 500 leaves the measurement area of the image processing apparatus 500, enters and leaves the measurement area of the image processing apparatus 501, and then enters a measurement area of the image processing apparatus 502 and then stands still in a sitting position. . In such a case, the integrated processing unit 503 processes the movement, position, and orientation of a person who is temporally continuous, and outputs the information as information 507.

In order to execute Examples 1 and 2 at the same time, it is possible to prepare a large number of image processing apparatuses 500 and perform spatial and temporal processing.

As a concrete application device of the above image processing device, a monitoring device for detecting human or environmental information from an image having a small number of pixels can be considered. When the number of pixels is not required so much, the image processing is easy and there is no problem in the amount of information to be detected, the smaller the number of pixels, the lower the cost and the more easily used. Further, the combined use of the visible image and the thermal image enhances the detection accuracy of human information in particular.

Also, in the care of a person such as medical care and nursing care, the state such as whether the person being cared for is awake, awake, awake, or asleep,
By using this image processing apparatus, a device that can be cared for even if the caregiver is not around for 24 hours becomes possible. It is also possible to detect the amount of clothing and the skin temperature, and judge whether the bedding is displaced or whether the body temperature is higher than usual.

A fourth embodiment of an air conditioner control apparatus using the image processing apparatus of the present invention will be described below.

FIG. 41 is a block diagram of the fourth embodiment. 6
Reference numeral 01 denotes a sensor unit, which is a two-dimensional infrared sensor 602 for measuring the temperature distribution in the room, a room temperature sensor 603, and a humidity sensor 6.
04 and an outside air temperature sensor 605. Reference numeral 606 denotes an image processing apparatus, and the two-dimensional infrared sensor 602 is shared with the sensor unit of the image processing apparatus 606. Reference numeral 610 is wind direction air volume information. In the comfort level detection unit 613, the foot position on the floor surface of each person in the room is detected by the image processing device 606. Therefore, if the wind direction and the air volume from the outlet are known, the air conditioner at an arbitrary position on the floor surface can be obtained. The airflow (m / s) can be calculated. Therefore, by obtaining wind direction air volume information 610 such as the angles of the upper and lower flaps and the left and right flaps that determine the wind direction and the rotation speed of the indoor unit fan that determines the air volume from the air conditioner, the air flow for each occupant at each position. The strength of is calculated. Reference numeral 611 denotes a control index determination unit, and an activity amount detection unit 612 and a comfort level detection unit 6 that calculates a comfort level PMV.
It is composed of 13. 614 is an air conditioning control unit, and 615
Is a calendar.

Next, the control index determination unit 611 will be described. The control index determination unit 611 according to the first embodiment calculates the comfort level for each person in the room and selects the control target for preferential air conditioning. In order to calculate the comfort level, the amount of activity is calculated and the amount of clothing is estimated for each person in the room as a human factor. Further, as environmental factors, room temperature, humidity, radiation temperature, and airflow are obtained from the environmental information extraction unit. Hereinafter, the activity amount detection unit and the comfort level detection unit will be described in this order.

It is known that when PMV is used as a control index as a comfort index as a control index, the amount of activity is an important factor as a human state. The amount of activity used in PMV is called the MET value, and the relationship between the MET value and the human state is
When lying down in a lying position and relaxing, the MET value is 0.
8. Quietly 1.0 while sitting on a chair, 1.4 when standing and relaxing, 1.6 when doing light work while standing, walking at 3 kilometers per hour It is 2.0 when the vehicle is running, and 3.0 when the vehicle is running at 5 kilometers per hour or when it is moving violently. Usually, if you have a standard life in your home, you will have an ME between 0 and 2.0.
Take a T value. In the conventional activity amount detection method, the movement of a heat-generating object in a certain area is detected by an infrared sensor, and the magnitude of the activity amount is determined by the magnitude of the output value. However, in the above-described conventional activity amount detection method, for example, even if there are a plurality of people in the room and some are stationary, if other people move around the room, the activity amount is uniformly large. There was a problem that the air conditioner was controlled according to the person who was judged to have a large amount of activity. In addition, the detection area of the infrared sensor for detecting the amount of activity is not only in a part of the room, but because the relationship between the output value of the infrared sensor and the actual movement of the person is ambiguous, the detection accuracy of the amount of activity is Was low. Therefore, in order to solve the above-described conventional problems, the present invention detects the position of an occupant in a wide range of the room, and obtains the stay frequency, which is the stationary duration with respect to the current position of each occupant, Detect the amount of activity for each person in the room.

FIG. 42 shows the activity amount detector 61 of the first embodiment.
2 shows a configuration diagram of No. 2. Reference numeral 620 is a personal information storage means for storing the foot position information of the occupants, 621 is a stay frequency calculation means for counting the stay frequency of each floor area divided into a plurality of areas in advance, and 622 is a stay frequency. It is an activity amount calculation unit that relates activity amounts. The operation of the activity amount detecting device configured as described above will be described.

When the room is a square room with 8 tatami mats, as shown in FIG. 43 (a) for detecting the position of a person, the floor surface is divided into 6 × 6 in units of 60 cm, for example,
Number it as a position number. Image processing device 6
The floor surface position for each occupant obtained by 06 is stored in the personal information storage unit 620 as personal information. Fig. 43 (b)
An example of personal information stored in the personal information storage unit 620 is shown in FIG. In the personal information, the measured time, the number of persons in the room at that time, and the floor number of each person in the room are described for the number of persons. Next, the stay frequency calculation means 621 counts the frequency of the visitor for each floor number from the floor position number in the personal information when using the foot position information for the past 3 minutes from the present time with a sampling cycle of 30 seconds. To do. Then, only the portion of the stay frequency corresponding to the floor number of the occupant obtained by the image processing apparatus 606 at the present time is extracted.

FIG. 44 (a) shows an example of the stay frequency corresponding to the floor surface position. It is considered that the stay frequency of the position number 32 is 6, and it is considered to be stationary, and the stay frequency of the person of the position number 17 is 1, and it is considered that the person has just moved from some other position. It can be considered that the person having the position number 9 has stayed at the position 9 and only two minutes have passed, or that he / she moved to another place for one minute during the three minutes. The stay frequency obtained here is an index indicating to what extent the occupant has been stationary in the past with respect to the floor surface position. That is, when a person is moving, the measurement interval is too long in the sampling period of 30 seconds and the moving distance of each person in the room cannot be calculated, and the moving distance of the person cannot be used as the activity amount. However, since the person is often stationary in the room rather than moving, it may be determined that the same person continues to stay if the person is in the same position even at a long time interval of about 30 seconds. In many cases.

The activity amount calculating means 622 associates the activity frequency with the stay frequency obtained for each staying person by the stay frequency calculating means 621. FIG. 44 (b) shows an example of the correspondence relationship. When the stay frequency is 6 which is the maximum value, the activity amount is 1.0 because it is considered to be stationary. As for the stay frequency of 5, it is possible that the user moved a little while moving still, so for the activity amount of 1.2 and the stay frequencies of 3 and 4, the transition status to the rest state or the stay frequency of other occupants was set. Since it may have been taken in, I assigned a high activity amount of 2 because it was a medium level of 1.6 and the stay frequency of 1 and 2 had just moved from another place. This correspondence changes depending on the number of samples stored in the personal information storage means 5. In this way, by successively calculating the stay frequency, in the case of a person with a high degree of stillness, when the stay frequency continues to output a value near the maximum frequency and often moves such as cleaning. Since the value with a low stay frequency continues to be output, the activity amount can be detected accurately. In addition, since it is possible to separate stationary people from active people, if there are people with high activity and people with low activity, the comfort level, which is a control index, is used as the activity level. It can also be adapted to people with low room.

Further, when the room is divided into regions, there are cases where people are always at the boundaries of the regions. In that case, if the motion of a person is a little and the foot position detection accuracy in the image processing device 606 is not very high, the detection position may shift each time the measurement is performed, even if the person is stationary. Therefore,
It is possible to accurately grasp a stationary person by predetermining a peripheral floor area for each floor position and considering the frequency of stays in the peripheral area when calculating the stay frequency. FIG. 45A shows an example of the peripheral definition for the floor surface.
Since the floor surface number 31 is at a corner, the floor surface numbers 25, 26, and 32 surrounding it are the periphery. The floor number 3 is located on the side, so the floor number 2 surrounding it is 2.
4, 8, 9, and 10 are defined as peripheral areas. Floor number 23
Is inside, so the floor numbers surrounding it are 16, 17, 1
8, 22, 24, 28, 29 and 30 are defined as peripheral areas. The stay frequency calculation means 621 counts the stay frequency of the floor number of the surrounding area in addition to the floor number of the current occupant. FIG. 45 (b) is an example of stay frequency calculation including the peripheral area for the personal information of FIG. 43 (b). The stay frequency of the person at the floor number 17 was 1 when the surrounding area was not included, but the stay frequency was 4 when the surrounding area was included, and the way of moving is not moving around in a distant place but is now. You can see that you are moving around the points, and you can relate a value that is not so high as the amount of activity.

Further, in the activity amount detection unit 613, when the sampling period of thermal image measurement is a short period of about 1 or 2 seconds, there is a limit to the moving speed of the person in the room. The moving speed can be obtained by associating the occupants from the extracted foot positions of the occupants (hereinafter, referred to as the moving amount). In this way, it is possible to improve the accuracy of activity amount detection by obtaining the amount of movement even for a person who is moving. The following is an example of activity detection using movement.

FIG. 46 shows a block diagram of the activity amount detecting section 613 using the movement amount of the first embodiment. 630 is continuous 2
A movement amount detecting means for calculating the movement distance from the foot position information of one measurement interval, 631 is a personal information storage means for storing information on the position and the movement amount in the past T1 minutes from the present time, 6
Reference numeral 32 is a period movement amount calculating means for obtaining a movement amount which is a representative of a predetermined period, 621 is a stay frequency calculating means for detecting a stationary person similar to that shown in FIG. 42, and 633 is an activity amount calculating means.

The activity amount detector 61 configured as described above.
The operation of No. 3 will be described. FIG. 47 shows a graph of a cleaning scene using a vacuum cleaner that seems to be most active indoors, which is recorded by video in an actual home and the moving distance per second is plotted with 60 cm as one unit. The horizontal axis is the elapsed time from the start of cleaning and is sampled every 1 second. The vertical axis represents the distance and one scale is 60 cm. The characteristic of cleaning is that the distance that the unit moves in a unit time, that is, the speed is not high, but that the movement of a short distance continues. The amount of movement is faster in the case of an activity such as going to get something in the room or when passing through the room, but it is often not active because it is completed in a few seconds. Therefore, when judging the amount of activity of a person in the room, rather than judging by a certain instantaneous travel distance,
It can be seen that it is suitable to make the determination by integrating the moving distance for a certain period. Now, regarding the operation of the activity amount detection unit 613, the obtained foot position information for each person in the room is sent to the movement amount detection means 630 as an input. The movement amount detecting means 630 will be described below.

FIG. 48 is a block diagram of the movement amount detecting means 630. Here, the sampling time is Δt, and the time t
The foot position information detected by the foot position information F (t), 1
The foot position information detected before the sample is used as the foot position information F
It is called (t-Δt). A position storage unit 640 stores the position information of the feet of the person in the room. 641 is the maximum value of the distance that a person can move within a predetermined time Δt, using the foot position information F (t) and the foot position information F (t-Δt) stored in the position storage means as inputs. It is a movable determination means that uses the maximum movement distance to search for an in-room person who is within the maximum movement distance of the foot position information F (t) from the foot position information F (t-Δt). A step 642 searches the foot position information F (t-Δt) output from the movement determining unit for the foot position information having the closest distance to the foot position information F (t), and determines two pairs of combinations in order. It is the nearest neighbor search means. Reference numeral 643 is a moving speed calculating means for inputting the distance output from the nearest neighbor searching means and outputting a moving amount for each person in the room.

Movement amount detecting means 6 configured as described above
The operation of No. 30 will be described. First, the foot position information detected by the image processing device 606 at each sampling timing is sequentially stored in the position storage unit 640. next,
The foot position information F (t) and the foot position information F (t−) stored in the position storage means 640 by the movable determination means 641.
Calculate the distance of Δt). At that time, distance calculation is performed for all combinations of t and t-Δt occupants, and a combination within the maximum movement distance is selected. Next, in the nearest neighbor search means 642, the foot position information F (t-Δt) and the foot position information F output from the movable determination means 641.
From the combinations of (t), the pair of combinations having the shortest distance is output. At this time, if the number of people increases during Δt, there will be some left in the foot position information F (t) that cannot be paired, but in this case, it is considered that entry from another room is the maximum. The moving distance is used as an alternative to the moving distance. Also, when the number of people decreases during Δt, there may be some left in the foot position information F (t−Δt) that cannot be paired, but in this case, since the number of people has decreased at this point, there is no problem.
Next, in the moving speed calculation means 643, the nearest neighbor search means 6
The moving speed of each occupant is calculated from the moving distance of each set and the time interval Δt output from 42. By calculating the amount of movement with the above configuration, even if a person is present at both corners of a room, for example, the correct movement speed of the person does not exceed the possible movement speed of the person. Can be asked. It is also possible to obtain the foot position by coordinates, calculate the movement distance from the coordinates, and obtain the movement amount without dividing the room into a plurality of regions. In this way, after obtaining the movement amount for each person in the room, a movement amount that is a representative of a certain period in the room (hereinafter, referred to as a period movement amount) is calculated. FIG. 49 shows an example of the personal information stored in the personal information storage means 631. In the personal information, the measured time, the number of persons in the room at that time, the floor number of each person in the room, and the distance are described for each person. The difference from FIG. 43B is that the distance is included. The period movement amount calculation means 632 associates the movement amount for each individual with the period movement amount. The period movement amount is expressed by ranks A, B, and C in descending order of activity. How to obtain Rank A Fig. 50
An explanation will be given with reference to (a). First, the average value of the movement amount of each occupant is calculated, and it is determined whether or not the threshold value H1 is exceeded. This process is performed for each measurement sample for T1 minutes, and the number of times in T1 minutes is counted. And
When the value is larger than the threshold value N1 of the predetermined frequency, the period movement amount rank A is set. Next, FIG.
How to obtain the period movement amount rank C will be described with reference to FIG. The frequency at which the average value of the movement amount falls below the threshold value H2 is counted for T1 minutes, and when the value is larger than the threshold value N2 of the predetermined frequency, the period movement amount rank C is set. If neither the rank A nor the rank C is entered, the period movement amount rank B is set. here,
The threshold is set not only for the amount of movement but also for the frequency, because it may not move into an active life scene or may move into a relaxing scene.

The activity amount calculation means 633 associates the period movement amount calculated by the above method with the presence or absence of stationary persons and the number of people obtained from the stay frequency calculation means 621 and the activity amount calculation means 633. FIG. 51 shows an example in which the period movement amount obtained from the period movement amount calculation unit 632, the stay frequency obtained from the stay frequency calculation unit 621, and the number of persons obtained from the image processing device 606 are related to the activity amount. In FIG. 51, the value in the table is the amount of activity, and the stationary person represents the case where there is an in-room person whose stay frequency is 90% or more of the maximum value in the stay frequency calculation means 621. By detecting the amount of activity by the above method, it becomes possible to detect the amount of activity according to the life scene.

Next, the comfort level detecting section 613 will be described. In the past, when calculating the comfort level, there was one that calculates the comfort level representing the room, but here, the comfort level is mainly calculated for each individual and these are used as control indices. Comfort PM
Environmental factors such as room temperature, humidity, and
Airflow and radiant temperature are required, and the amount of activity and the amount of clothing are required as human factors. Also, P for calculation
An MV calculation formula is required. For each element, the room temperature uses the measured value of the room temperature sensor 603 that measures the intake temperature of the air conditioner, and the humidity uses the measured value of the humidity sensor 6044 installed in the main body of the air conditioner. The air flow is obtained from the wind direction air volume information and the foot position for each individual obtained from the image processing device 606. The radiation temperature is also obtained from the image processing device 606. The activity amount for each individual is obtained from the activity amount detection unit 612. The amount of clothing is the image processing device 606.
Get from In that case, since clothes are often changed depending on the outside temperature and the season, each month is made to correspond to the four seasons of spring, summer, autumn, and winter based on the value of the calendar 615, and the rank is conditioned on the value of the month and the outside temperature. It is also possible to divide and correct. This makes it possible to detect the clothing amount using the thermal image of the target person. An example of the PMV calculation formula is shown below.

PMV = (0.303e-2.1M ⁺ 0.028) * {58.
15 (MW) -3.05 · 10 ^-3 [5733-406.7 (MW)-
Pa] -24.42 [(M-W ) -1] -10 -3 · M (5867-P
a) -0.0814 ・ M (34-ta) -3.96 ・ 10 ^-8 fcl [(t
cl + 273) ⁴ − (tr + 273) ⁴ ] −fcl · hc (tcl−t
a)} Here, the meaning of each symbol is as follows. M: Metabolism (Met value) W: Mechanical work to the outside (can be assumed to be zero for most of the metabolism) Pa: Water vapor pressure ta: Air temperature tr: Average radiation temperature fcl: Part covered with clothes And the area ratio of exposed parts tcl: Surface temperature of clothes hc: Convection heat transfer coefficient The convection heat transfer coefficient hc satisfies the condition of 2.38 (tcl-ta) ^0.25 > 12.1kgVr when Vr is an air flow (m / s). For example, hc = 2.38 (tcl-ta) ^0.25 , and when 2.38 (tcl-ta) ^0.25 ≤ 12.1 kgVr, it can be determined that hc = 12.1 kgVr.

As for the area ratio fcl of the area covered with clothes and the area exposed, fcl = 1.00 + 0.2.Ic
l If the condition of Icl ≧ 0.5 is satisfied, fcl = 1.05 + 0.1 · Ic
can be defined as l.

As described above, since each input element for calculating the PMV can be detected, the comfort level PMV for each individual can be calculated. Next, the control index for controlling the air conditioner is determined based on the comfort level calculated for each individual. When there are multiple people in the room, the comfort level of each individual may differ depending on the indoor and outdoor conditions such as the activity level and the influence of spatial radiation. In that case, when there is an in-room person whose comfort level detected by the above method is a predetermined comfort zone, for example, PMV ± 0.5, the in-room person is preferentially air-conditioned. In addition, when all the people in the room enter the comfortable area, the wind direction control is performed centering on the position of the person in the room.

An example of air conditioning control in the fourth embodiment will be shown below. FIG. 52 (a) shows an example of air conditioning during heating in a stable state where there are two people in the room and their PMVs are almost the same. 650 is indoors, 651 is an air conditioner,
652 is an infrared two-dimensional sensor, 653 is a sitting person,
654 is another sitting person, 655 is a door, 65
6 is the wind direction and volume, and 657 is the detection range of the two-dimensional infrared sensor. In this case, the occupant 653 and the occupant 654 have the same amount of activity because they are still in the sitting position, and the floor surface,
Assuming that the radiation received from the wall surface and the strength of the airflow are also equal, the PMVs of the two persons are the same, the blowing temperature is wide enough to maintain the indoor thermal environment, the wind direction is wide, and the air volume is medium. FIG. 52B is an example of air conditioning when time passes from the state of FIG. 52A, the occupant 658 moves, the door of the room opens, and cold air enters the room. 658 is a person who has moved to a standing position, 659 is a person who is still in a sitting position, 660 is an open door, 66
1 is the wind direction and volume at this time. In this case, the door 66
Due to the cold air entering from 0, the floor temperature near the occupants 659 is cooled, and the temperature drop on the floor and walls is caused by the two-dimensional infrared sensor 652.
Observed by.

Further, since the person in the room at 658 starts to move in the standing position, a higher amount of activity is observed as compared with the person in the room at the sitting position at 659. In this case, since the PMV is high near the occupant 658 and low near the occupant 659, the wind direction is 6
59 is centered. Further, the blowing temperature is controlled to be high because cold air has entered through the door 660. In addition, even if the occupant 658 does not move, the door 660
Even if the temperature of the floor or wall near the occupant 659 decreases due to the cold air from the room and the radiant temperature to the occupant 659 is low, the wind direction is directed toward the occupant 659, but since the activity amount is the same, then the door 660 is used. If is closed, it returns to stable air conditioning control in a short time. In this way, by estimating the comfort level of each person in the room and performing air conditioning based on that, it is possible to perform fine air conditioning in accordance with the comfort level of the person. If there is no person in the room, it is assumed that there is a person in the center of the room with zero activity and a predetermined amount of clothing, and a control index is set, or the place where the temperature of the floor or wall in the room is low, Alternatively, control such as blowing hot air to a high place can be performed. In the case of cooling, the same control can be performed.
Further, in the case of controlling the wind direction, if it is unstable to change the wind direction for each sample, the position movement of the person in the room is checked while observing the output regarding the foot position information of the image processing device 606, and the position is continuously determined at the same position. It is also possible to target the wind direction control only when there is a person in the room.

Next, an embodiment of a control device for an air conditioner in which a life scene estimation unit is added to the above-mentioned Embodiment 4 will be shown as Embodiment 5. FIG. 53 is an overall configuration diagram of the fifth embodiment. The difference from FIG. 41 is that a life scene estimation unit 670 is added to the control index determination unit 611, an optical sensor is added to the sensor unit 601, and a clock is added.

The life scene estimation unit 670 will be described below. In the life scene estimation unit 670, the activity amount detection unit 6
The amount of activity for each individual obtained from 12 and the image processing device 60
The indoor life scene is estimated from the number of people obtained from No. 6, the illuminance level in the room obtained from the optical sensor 671, and the time obtained from the clock 672. Here, the life scene is a name when a living activity or a living condition performed by an occupant is viewed in an indoor unit, and is generally called by a name such as drank or relaxing. If there is one person,
The person's life act becomes a life scene, but in the case of multiple cases, it must be estimated in consideration of the activity level of each person. Estimate life scenes such as leaving, entering, sleeping, waking up,
There were 6 transitional scenes, one for hot people and one for cold people, and three stable scenes for relaxing, gathering, and active housework. The transitional scene here is a scene in which the PMV has no validity as a control index, and the stable scene is a scene in which the occupant stays in the room for a certain period of time and the PMV can use it as a control index. is there.

The estimation method of each scene will be described below. First, we describe the method of estimating transient scenes. As for the leaving scene, when the number of persons obtained from the image processing device 606 becomes 0, the absence continuation time starts to be counted, and when the number of persons is 0 or more and continues for 10 minutes or more, it is determined as the leaving scene. Here, the absence continuation time is usually counted because the room is often in contact with the next room or the outside and the room is moved.Therefore, it is the control to change the control index by judging the exit for each short absence. Is unstable, the exit scene is judged after waiting a certain period of time in order to divide the scene into one segment. Next, the entry scene is assumed to be the entry scene when the current life scene is the exit scene and one or more people are detected, and the entry scene is held for 20 minutes. Next, in the sleeping scene, when the illuminance obtained from the optical sensor is less than or equal to the predetermined brightness, the time is from 9:00 pm to 6:00 am, and the amount of activity is 1.0 or less for 30 minutes. If it is, it is determined to go to bed. Also, if the current life scene is a bedtime scene and the amount of activity is 1.2 or more for 5 minutes,
Alternatively, when the illuminance exceeds a certain value, it is determined that the scene is a wake-up scene, and is held for 20 minutes. In the hot occupant scene and the cold occupant scene, the skin temperature obtained from the image processing device 606 is
Judgment is made based on the fact that the standard skin temperature range of the human body, which is obtained from the calendar in advance and is determined for each month, is exceeded.

Next, a stable scene will be described. As for the relaxation scene, if the activity level rank of 1.0 or less continues for 10 minutes, and if the activity level of 1.2 to 1.7 continues for 10 minutes, For housework, each scene judgment condition is when the amount of activity is 1.8 or more for 10 minutes.

The air-conditioning control index is set in advance for the life scene estimated as described above. less than,
An example of the control index determined for each scene will be described. Here, the standard target comfort level is 0. In the case of leaving the room, the target comfort level is lowered to -1.0 in the case of heating, and the wind direction is directed to the center of the room. When entering a room, set the target comfort level to 1.0,
Turn the wind toward the occupants. In the case of a sleeping scene, the target comfort level is decreased by 0.5 every two hours, and the wind direction avoids the human body, and the air volume is set to a light breeze. In the case of a wake-up scene, the target comfort level is set to 2.0 and the wind is directed toward the occupants. In the case of a hot room scene during cooling, lower the target comfort level to -2.0,
Cool in the wind direction centered on people with high skin temperature. In the case of a cold occupant scene during heating, the target comfort level is raised to 2.0 and heating is performed in the wind direction centering on the occupants with low skin temperature. However, it is not good for the health to direct cold air or hot air to the human body for a long time. Therefore, the hot occupant scene and the cold occupant scene are limited to 3 minutes. In the case of a relaxing scene, the target comfort level is set to 0.5 and slightly warm. The target comfort level is set to -0.5 for a casual scene and -0.5 for an active housework scene. In this way, by performing air conditioning according to each scene, it is possible to air conditioning according to the actual conditions of life.
If a stable scene is estimated, the control index is
With the comfort level described in the embodiment, it is possible to switch to the above-mentioned control index for each scene in the case of a transient scene. The air conditioning control unit 614 obtains these control indexes,
Control equipment.

The image processing apparatus or the activity amount detecting unit 612 of the above-mentioned Embodiments 1 to 3 detects any one of the presence / absence and number of persons in the room, foot position, posture, clothing amount, skin surface temperature, and activity amount. It is also possible to use it in the application equipment of the image processing apparatus for the purpose of For example, a crime prevention device that detects an intruder using the human area detection unit of the present embodiment. An apparatus for detecting the degree of congestion in a room using a number-of-people detecting means. Further, a device for controlling environment such as lighting and sound according to a position of a person, which uses a foot position detecting means or an activity amount detecting section. A device for detecting the condition of a patient or a sick person used for medical treatment or nursing, which uses a posture detecting unit, a clothing amount detecting unit, and a skin temperature detecting unit, can be considered.

The sixth embodiment of the present invention will be described below. FIG. 54 is a configuration diagram of a control index determination unit according to the sixth embodiment of the present invention. The other structure is the same as that of the fifth embodiment shown in FIG. Reference numeral 680 is an absent control index determination unit that outputs a control index when there is no person in the room. FIG. 55
Shows the structure of the absence control index determination unit. An occupancy data storage unit 690 determines the presence or absence of an occupant (hereinafter, referred to as occupancy data) from the number of persons obtained from the image processing apparatus 606, and stores it as time-series data. Reference numeral 692 is a base temperature converting means. here,
The base temperature is the temperature level that is maintained at a minimum, and differs between heating and cooling.

The operation of the absence control index determining unit 680 will be described below. The occupancy data storage unit 690 detects the presence or absence of a person from the number of persons obtained from the image processing apparatus 606 as 1 when the occupant is present and 0 when the occupant is not present. The data of 0 and 1 vary greatly depending on the presence / absence of people entering and leaving the room.
0 minutes as 1 unit, for example 10 minutes or more, the number of people is 0
Set to 0 only for humans, and 1 otherwise. Further, it is possible to perform data compression or the like in order to reduce the storage capacity of the occupancy data storage means 690. Occupancy data storage means 690
Then, data management is also performed so that one day's occupancy data can be stored for a certain number of days dating back from the current day. The reason why the old data is not used for a certain period from the day of use is that using old data for calculating the occupancy rate, which is required later, is unlikely to reflect recent living conditions. Here, the upper limit is 60 days. Next, the occupancy pattern classifying unit 691 collects occupancy data on the same day and on the same day attribute based on the date attribute information from the calendar, and obtains the occupancy rate by time. Fig. 56 (a) shows the occupancy rate on weekdays in the living room of an actual home, as shown in Fig. 56 (b).
The holiday occupancy rate is shown in. The horizontal axis represents time, starting at 4:00 am and the unit is 30 minutes. The vertical axis is the occupancy rate, which is obtained by dividing the frequency of staying in a certain time zone by the number of days of all weekdays or all holidays, and then multiplying by 100. As can be seen from these figures, the pattern of occupancy varies with the attributes of the day. Therefore, the occupancy pattern is classified here by the attribute of the day.

FIG. 57 shows an example of the occupancy pattern classification processing. 700 is occupancy data stored in the occupancy data storage unit 690, 701 is a calendar, 702 is day information obtained from the calendar 701, and 703 is an occupancy pattern for each day of the week. For example, if the current day is Wednesday, only the Wednesday data in the occupancy data stored in the occupancy data storage unit 609 is extracted, and the frequency is counted for each time period. Then, the occupancy rate by time is obtained by dividing the frequency by time by the number of days on all Wednesdays. Then, it is stored as a room presence pattern for Wednesday.

Next, the base temperature converting means 692 calculates the value of the base temperature based on this occupancy pattern. FIG. 58 shows the relationship between the room presence pattern and the base temperature.
710 is the room-occupancy rate by time, 711 is the base temperature that is the output, and 712 is the standard set temperature when the user is in the room. The horizontal axis of the graph is time, the left vertical axis is the room occupancy rate, and the right vertical axis is the temperature.
The base temperature conversion method is such that the range of possible values of the base temperature is the lower limit p degrees to the upper limit q degrees, the current time is t, the base temperature at t is y (t), and the occupancy rate at t is z (t). In this case, y (t) = (q−p) × z (t + 1) + p. However, the unit of time is one hour. Fifth
In the case where the life scene estimation unit 670 shown in the above example shows the exit scene and the person in the room is absent, the base temperature value for each time is set as a target value and the base temperature is changed according to the schedule. Then, when the entry of a person is detected, the target value is switched to the control index when there is a person in the room. By reflecting the occupancy rate in the base temperature in this way, it is possible to perform a standby before the person enters the room environment, and it is possible to set a lower base temperature during times when there are many absences, so it is economical. Even excellent air conditioning can be realized. Also, it is not necessary to use a timer except in special cases.

In the present embodiment, the day of the week is the basic classification attribute of the occupancy pattern. However, the occupancy data having a similar tendency is pattern-classified from all the occupancy data, and the attribute of the day classified into the patterns is obtained. It is also possible to obtain the feature of from the occupancy data. Furthermore, excluding the image processing unit, using the output of the human body detection sensor that detects only the presence or absence of the human body, the presence or absence of the room is determined, and the control index determination unit for presence of the room to determine the control index when the presence of the room, It is also possible to adopt a configuration including an unoccupied air-conditioning control index determination unit having the room presence pattern classification and base temperature conversion. In this case, the in-room control index may be temperature or comfort level. In addition, the comfort level PMV can be maintained by using the minimum comfort level when not at home, instead of the base temperature.
It is also possible to use as a control index.

The seventh embodiment of the present invention will be described below. In the seventh embodiment, as shown in FIG. 59, a remote controller learning unit 720 and a remote controller 721 are provided between the control index determination unit 611 and the air conditioning control unit 614 of the fourth embodiment, and the remote control operation regarding the user's control index is performed. Is learned as the user's preference, and the control index calculated by the control index determination unit 611 is corrected.

FIG. 60 is a block diagram of the remote controller learning unit 720 of the seventh embodiment of the present invention. In FIG. 60, reference numeral 731 indicates a learning means. There are various methods for the learning means 731, but in this embodiment, a neural network is taken as an example. Reference numeral 732 is a comfort level correction means. 733 is the comfort level calculated by the comfort level detection unit 613, 734 is the life scene estimated by the life scene estimation unit 670, 735 is the time and calendar, and 736 is the operation panel obtained from the remote controller that the user has declared the comfort level. An operation signal, 737 is a comfort level correction value, and 738 is a target comfort level as a final control index.

The operation of the configuration shown in FIG. 60 will be described below. The calculated comfort level 733 obtained from the control index determination unit 611 and the remote controller 721,
Life scene 734, time / calendar 735, operation panel operation signal 73 from the remote controller that declared the comfort level of the user
Any one of 6 is input as an input signal to the neural network which is a constituent element of the learning means 731. The neural network infers the operation of the remote controller for which the user has declared the comfort level. In other words, for the comfort level calculated by the control index determination unit 611, for example, the user is "hot",
It is judged that the declaration of "cold" or the like is the user's preference, and when similar environmental factors or human factors are observed, the control index is corrected based on the content of the user's preference. is there. The user comfort level correction value 737 output from the neural network is input to the comfort level correction unit 732, and the calculated comfort level 733 is calculated to generate the target comfort level 738. The calculation method is to add the simplest method, and when it becomes complicated, the comfort level correction value 737 and the calculated comfort level 7
It is also possible to use a linear or non-linear function that takes 33 pieces of information as arguments. The learning means 731 uses the input value 733 of each means.
To 735 are used as a teacher signal for the remote control operation signal 736 to learn the neural network. After learning, the state in the neural network is changed. Although various patterns of the pattern classification type can be used as the neural network, in the present embodiment, LVQ (learning vector quantu) is used.
m: References, T. Kohonen, "Self-Organization and Asso
c-iative Memory ", 2ed, Springer-Verlag, 1988) as an example. The forward calculation of the neural network used in this embodiment will be described based on FIG.
740 is an input signal normalization section, and 741 is a reference information section, which has three categories according to human intention and m pieces of reference information in each category. Reference numeral 742 is a distance calculation calculation unit, and 743 is a category calculation unit. The operation of the configuration shown in FIG. 61 will be described below. The input signal 744 is normalized by the input signal normalization unit. That is, in the input signal normalization unit 740, Sx1 = (xmax−x1) / (xmax−xmin) (1) Sx1: normalized signal value xmax: maximum value of input signal xmin: minimum of input signal The process represented by the value expression (1) is performed on all input signals. Input signals include temperature, comfort level, life scene, time of day, and the like.
The signal normalized by the input signal normalization unit 740 is output to the distance calculation unit 742 as the input vector 745.
In the distance calculation unit 742, the input vector and reference information unit 74
The distance from the reference information of each category of 1 which represents the intention of the human being, which is hot, cold, and satisfied, is calculated. That is, the distance (dAj) from the reference information of the hot category is obtained by the equation 2.

DAj = Σ (xi-Raji) ² (2) dAj: Distance from jth reference information xi: i-th input vector value Raji: i-th vector value of jth reference information Reference information By referring to the directory 746 of the section 741, the number of categories 3 and the number of reference information of each category are calculated.

The distance between the reference information and the input vector is calculated for each category, and the same process is performed for all categories to calculate the distance. The total number (dnum) of distances calculated by the distance calculation unit 742 is dnum = m1 + m2 + ... + mN (3). Next, the distances dNj and d calculated by the distance calculation unit 742
The num is output to the category calculation unit 743. In the category calculation unit 743, the shortest distance d out of dnum distances.
min is calculated, and the category to which the corresponding reference information belongs is calculated.

The category to which the reference information having the shortest distance to the input vector belongs is output from the category calculation unit 743.
That is, it becomes the output of the neural network which is a component of the learning means 731 in FIG. This output is subjected to calculations such as being added to the comfort level calculated by the comfort level detection unit 613 as a comfort level correction value 737 in the comfort level correction unit 732.

The learning of the neural network will be described in detail with reference to FIG. In FIG. 62, an input normalization unit 750 normalizes an input signal common to the input signal normalization unit 740 of FIG. 61 and converts it into an input vector. The reference information section 751 is also a reference information section common to FIG. Reference numeral 752 is a reference information search unit, 753 is a reference information generation unit, and 754 is a reference information learning unit. The operation of the above configuration will be described below.

The input signal normalized by the input signal normalization unit 750 is used as an input vector by the reference information search unit 752.
Entered in. Furthermore, the correct category (teacher data) of the input vector is also input at the same time. Reference information search unit 7
In 52, the directory of the reference information section 751 is referred to,
If the correct category (TC) of the input vector is, for example, a hot category, and the number of reference information items in the hot category has the following relationship, the following process is performed.

If ARNUM <ARNUMmax, the processing of the reference information generating section 753 is performed.

If ARNUM ≥ ARNUMmax, the processing of the reference information learning unit 754 is performed. ARNUM: number of reference information of hot category ARNUMmax: maximum number of reference information of A category The reference information generation unit 753 generates an input vector as reference information of a correct category (TC). Here, the reference information of the hot category is generated and ARNUM is incremented by 1.
The reference information learning unit 754 compares the output value of the remote control operating means with the correct category of the input vector, and if they match, the reference information with the shortest distance is brought closer. If the output value of the remote control operating means does not match the correct category of the input vector, the reference information with the shortest distance is moved away. That is, if OC = TC, then RV = RV + α (R
V-Sx) If OC ≠ TC, then RV = RV-α (R
V-Sx) RV: Reference vector with the shortest distance from the input vector OC: Output category (RV category) TC: Correct category of input vector Sx: Input vector α: Learning rate Therefore, the reference information learning unit 754 has reference information. Until the number exceeds the maximum value of the reference information, the input vector is generated as the reference information of the correct category of the input vector. Also,
After the maximum value is exceeded, when the output category and the input category match, the reference information is moved a little closer to the direction of the input vector, and when they do not match, the reference information is moved a little away from the input vector. The distance to approach or move away is set by the learning rate α.

By learning the remote control operation by the neural network as described above, the comfort level and the control index for each life scene can be corrected by changing the combination of the input data. It becomes possible to realize air conditioning. Also, by installing a remote control position sensor in the remote control and checking the correspondence between the foot position information and the remote control position sensor and using it as an input element of the neural network, the correction of the control index for each individual or the correction of the control index for each position of the human body is corrected. It is also possible to do

The eighth embodiment of the present invention will be described below. The eighth embodiment relates to a traffic flow control device using the image processing device of the present invention. The traffic flow here is a flow of people involved in transportation or traffic, and includes an elevator, a traffic light, and a transportation vehicle. There is a conventional traffic flow control means that detects a change in the elevator car load and predicts and controls the traffic flow, but this method delays the number of people detection, and the control becomes a follow-up control. There were many things. The present traffic flow control device accurately detects the number of people waiting for transportation by distinguishing them from passersby, and controls the traffic flow using the result. FIG. 8 shows a block diagram of the eighth embodiment. Reference numeral 800 denotes an image processing apparatus, 801
Is a traffic flow classification means, 802 is a traffic flow prediction means, and 803 is a traffic flow control means. With the above configuration, the number of people, especially the number of people waiting for a vehicle, an elevator, or the like is obtained from the image processing device 800. Next, traffic flow classification means 8
In 01, the feature quantity relating to the increase / decrease in the number of people is extracted from the time-series data of the number of people obtained from the image processing apparatus 800, and pattern classification is performed. In other words, it is classified into which time zone it is crowded. In the traffic flow predicting means 802, the image processing device 800
Predict the traffic flow from the current number of people and traffic flow classification pattern obtained from. For example, in the case of an elevator, the time zone and the number of floors that are likely to be congested are predicted. Traffic flow control means 803
Then, based on the predicted traffic flow, for example, the operation of a plurality of elevators is optimally controlled so that the waiting time of the person on each floor is reduced.

[0154]

According to the image processing apparatus of the present invention, the inside of the position determination unit is divided according to the type of posture in the foot position detection unit according to the number of determined postures such as standing posture and sitting posture. The determination is performed, and the output from the position determination unit is selected and selected according to the output from the posture determination unit. By performing such processing, more accurate position detection becomes possible.

Further, since it is difficult to detect the environmental information in the detection area, for example, structural information such as the size of the room or the position of the wall surface with respect to the sensor in the case of a room from the thermal image,
The user can directly input from the console unit and can be used as auxiliary information for accurate position detection and posture detection.

Next, the image processing apparatus according to the present invention is not limited to the processing of the thermal image, but the system using the visible image input device (camera) is used to display the relevant information of the person obtained from the thermal image on the visible image. It can be communicated to provide the user with the location of the person in the visible image.

In the opposite case, it is possible to detect the position of the person on the thermal image by using the information of the person obtained from the visible image, for example, the motion information, and the processing of the thermal image and the visible image can be performed. The performance of the neural network in the image information detection unit, which is being performed, can be adaptively learned by using mutual information as teachers.

By using a plurality of image processing apparatuses in which the thermal image processing and the visible image processing of the present invention are combined, it is possible to more accurately detect the information of the person in the area where the plurality of apparatuses are installed. Although it is difficult for one image processing apparatus to accurately separate overlapping persons, it is possible to use a plurality of apparatuses. In addition, people can be tracked in a large area.

As described above, the control device for an air conditioner according to the present invention includes an image processing device having infrared image input means for measuring the temperature of the person in the room and the environment, and personal information obtained from the image processing device. Also, it has a control index determination unit that determines the control index for air conditioning from the environmental information and an air conditioning control unit that controls the air conditioning equipment, and considers various environmental conditions. A comfortable air conditioning can be realized according to the position and condition of the individual. Further, since the personal information and the environmental information are extracted by performing image processing on the thermal image, it is not necessary to dispose a plurality of sensors in the room, and it is easy to implement.

Further, by adding an absence control index determining unit consisting of the occupancy data storage means, the occupancy pattern classification means, and the base temperature conversion means to the above configuration, the timer can be eliminated and comfort can be improved. Air conditioning operation with economical efficiency is possible.

Furthermore, by adding a remote control operation learning section to the above configuration, it becomes possible to perform air conditioning control that reflects the user's preference.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2A is a diagram showing an example of a thermal image in the same embodiment. FIG. 2B is a diagram showing an example of a plan view in the same embodiment.

FIG. 3 is a configuration diagram of a human area detection unit in the embodiment.

FIG. 4 is a diagram showing an example of block division of a thermal image in the embodiment.

FIG. 5 is a configuration diagram of a human area detection unit using a difference image in the example.

FIG. 6 is a diagram showing an example of creating a difference image in the embodiment.

FIG. 7 is a configuration diagram of a personal information extraction unit in the same embodiment.

FIG. 8 is a configuration diagram of the number-of-people feature detecting means in the embodiment.

FIG. 9 is a diagram showing an example of determining the number of people in the same embodiment.

FIG. 10 is a block diagram of a person number detecting means using a neural network in the same embodiment.

FIG. 11 is a configuration diagram of detecting a foot position in the embodiment.

FIG. 12 is a schematic diagram of a thermal image of a standing human.

FIG. 13 is a diagram showing a method of calculating a position determination feature amount in the embodiment.

FIG. 14 is a configuration diagram of posture detection in the embodiment.

FIG. 15 is a diagram showing a processing example of a posture detection unit in the embodiment.

FIG. 16 is a diagram showing a procedure of standardizing human pixel blocks in the embodiment.

FIG. 17 is a diagram showing a method of calculating a posture determination feature amount according to the embodiment.

FIG. 18 is a partial configuration diagram of an image information detection unit in the embodiment.

FIG. 19 is a configuration diagram of a personal information extraction unit in the same embodiment.

FIG. 20 is a configuration diagram of skin temperature detection in the example.

FIG. 21 (a) is a diagram showing an example of block division of a standing human in the same embodiment. FIG. 21 (b) is a diagram showing an example of block division of a sitting human in the same embodiment.

FIG. 22 is a configuration diagram of detection of a clothing amount in the example.

FIG. 23 is a configuration diagram of an environment extraction unit in the same embodiment.

FIG. 24 is a configuration diagram of a foot temperature detecting means in the embodiment.

FIG. 25 is a diagram showing a processing example of a foot temperature detecting means in the embodiment.

26 (a) is a diagram showing a thermal image of each part of the room in the case of the central installation in the same embodiment. FIG. 26 (b) is a diagram showing a thermal image of each part of the room in the case of the left installation in the same embodiment.

FIG. 27 is a configuration diagram of radiation temperature detection means in the same embodiment.

FIG. 28 is a block diagram of a different environment extraction unit in the same embodiment.

FIG. 29 is a configuration diagram of a different image processing unit in the embodiment.

FIG. 30 is a configuration diagram of a human area correction unit in the embodiment.

FIG. 31 is a configuration diagram of a cumulative processing unit in the embodiment.

FIG. 32 is a diagram showing an example of position accumulation in the example.

FIG. 33 is a diagram showing a relationship between cumulative storage A and cumulative storage B in the embodiment.

FIG. 34 is a diagram showing a method of determining an air conditioner installation position in the same example.

FIG. 35 is a diagram showing a method of determining an indoor representative point in the same example.

FIG. 36 is a configuration diagram of an image processing apparatus according to a second embodiment of the present invention.

FIG. 37 is a configuration diagram of an image processing apparatus according to a third embodiment of the present invention.

FIG. 38 is a configuration diagram of an image processing apparatus in which a monitoring information detection unit is added to the image processing apparatus according to the third embodiment of the present invention.

FIG. 39 is a configuration diagram of an image processing apparatus in which a personal identification unit is added to the embodiment.

FIG. 40 is a configuration diagram of an integrated processing unit that integrates outputs of a plurality of image processing apparatuses of the present invention.

FIG. 41 is a configuration diagram of an air conditioner control device using an image processing device according to a fourth embodiment of the present invention.

FIG. 42 is a configuration diagram of an activity amount detection unit in the same example.

43A is a diagram showing floor surface position division in the embodiment, and FIG. 43B is a diagram showing an example of stored personal information in the embodiment.

FIG. 44 (a) is a diagram showing a stay frequency in the same example. (B) is a diagram showing a correspondence between a stay frequency and an activity amount in the same example.

FIG. 45 (a) is a diagram showing a definition example of the periphery of the floor surface in the same embodiment. FIG. 45 (b) is a diagram showing a stay frequency calculation example including the periphery in the same embodiment.

FIG. 46 is a block diagram of an activity amount detection unit using movement amounts in the same embodiment.

FIG. 47 is a diagram showing an example of a moving distance in an actual cleaning scene.

FIG. 48 is a block diagram of a movement amount detecting means in the embodiment.

FIG. 49 is a diagram showing an example of stored personal information in the embodiment.

50A is a diagram showing a method of deriving a period movement amount rank A in the same embodiment. FIG. 50B is a diagram showing a method of deriving a period movement amount rank C in the same example.

FIG. 51 is a diagram showing a correspondence between a period movement amount and the activity amount in the same example.

FIG. 52 (a) is a diagram showing an example of stable air conditioning in the same embodiment. (B) is a diagram showing an example of air conditioning when conditions in the same example are changed.

FIG. 53 is a configuration diagram of a control device for an air conditioner in a fifth embodiment of the present invention.

FIG. 54 is a configuration diagram of a control index determination unit in the sixth embodiment of the present invention.

FIG. 55 is a configuration diagram of an absence control index determination unit in the example.

[FIG. 56] (a) A diagram showing the percentage of occupancy by time of day in a real home (b) A diagram showing the percentage of occupancy by time of day in a real home

FIG. 57 is a diagram showing an example of classification of occupancy patterns in the example.

FIG. 58 is a diagram showing the relationship between the room presence pattern and the base temperature in the example.

FIG. 59 is a configuration diagram of a control device for an air conditioner according to a seventh embodiment of the present invention.

FIG. 60 is a configuration diagram of a remote control learning unit in the example.

FIG. 61 is a configuration diagram of forward calculation of the neural network in the example.

FIG. 62 is a configuration diagram of learning of the neural network in the example.

FIG. 63 is a configuration diagram of a traffic flow control device according to an eighth embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 sensor part 2 image processing part 3 image information detecting part 460 console part 470 visible sensor part (camera) 471 visible image processing part 472 visible image information detecting part 480 monitoring information detecting part 503 integrated processing part 611 control index determining part 612 activity amount Detection unit 613 Comfort level detection unit 614 Air conditioning control unit 670 Life scene estimation unit 680 Absence control index determination unit 720 Remote control operation learning unit

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Masaaki Sato Inventor Masaaki Sato 3-10-1 Higashisanda, Tama-ku, Kawasaki-shi, Kanagawa Matsushita Giken Co., Ltd. (72) Inventor ▲ Kunio Tada Higashi, Tama-ku, Kawasaki-shi, Kanagawa Mita 3-10-10 Matsushita Giken Co., Ltd. (72) Inventor Ikuo Akamine 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Ma, Shimizu 1006 Kadoma, Kadoma City, Osaka Matsushita Electric Industrial Co., Ltd. Sangyo Co., Ltd. (72) Inventor Yoshiaki Uchida 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd.

Claims (44)

[Claims] 1. An infrared image (hereinafter referred to as a thermal image) for measuring the temperature of a person or environment in at least a detection area.
A sensor unit having an input unit, an image processing unit for extracting a feature amount from a thermal image having a detected temperature as an output of the sensor unit as a pixel value, and human personal information based on a signal from the image processing unit. An image processing apparatus comprising an image information detection unit that outputs a signal or an environmental information signal. 2. An image processing section for storing at least a thermal image measured by an image processing section, and a human area detection section for detecting an area of a human part in the thermal image and outputting a human pixel block and its representative point. 2. The image processing apparatus according to claim 1, further comprising: a personal information extraction unit configured to extract a human feature amount in the detection area based on the human pixel block detected by the human region detection unit. 3. The image information detecting section has a neural net or a pattern recognizing mechanism as a constituent element, and at least a number of people detecting means, a foot position detecting means, a posture detecting means, a clothing amount detecting means, and a skin temperature detecting. The image processing apparatus according to claim 1, further comprising any one of the units. 4. The image processing apparatus according to claim 2, further comprising an environment information extraction unit that extracts environment information in the detection area from the measurement image, to the image processing unit. 5. A human region correction unit that complements and erases a human pixel block obtained by the human region detection unit at the pixel level in an image processing unit, and a representative of the human pixel block obtained by the human region detection unit. An accumulation processing unit that accumulates points for a predetermined period, and at least a floor and a wall, or an indoor representative point that is a boundary between walls, is determined from the accumulation of representative points of human pixel blocks obtained from the accumulation processing unit for a predetermined period, The image processing apparatus according to claim 2, further comprising an indoor part determination unit that associates each pixel on the thermal image with the correspondence between the indoor floor and the wall. 6. A human region detection unit divides a measured thermal image into a plurality of regions and calculates a representative temperature of each region, and a region division representative temperature calculating unit, and a human body cutout temperature for each representative temperature of each region. A cutout temperature determining unit, a human detection unit that cuts out a thermal image with a cutout temperature width determined for each region, and a representative point calculation unit that calculates a representative point of the cut out human pixel block. The image processing apparatus according to any one of claims 2 to 5. 7. A feature amount for the personal information extraction unit to detect the number of people from the thermal image obtained from the image storage unit, the human pixel block obtained from the human region detection unit, and the representative point of the human pixel block. A feature quantity for determining the posture of a person in the detection area is detected at least from the number of persons feature detecting section to be detected, the thermal image, the representative point, and the number of persons obtained from the number of persons feature detecting section or the image information detecting section. Posture determination feature amount detection unit, position determination feature amount detection unit that detects a feature amount for determining the same person's position, and clothing amount determination feature that detects a feature amount for determining the same person's clothing amount 7. The amount detection unit and the skin temperature determination feature amount detection unit for detecting the feature amount for determining the skin temperature of the same person, as claimed in any one of claims 2 to 6. Image processing device. 8. The number-of-people feature detecting unit has a human pixel block obtained from the human region detecting unit, a pixel number calculating unit for counting the number of pixels of the human pixel block from a representative point, and a form of the human pixel block in advance. The image processing according to any one of claims 2 to 7, further comprising: a block form determination unit that classifies the blocks into predetermined form types and a human pixel block counting unit that counts the number of blocks of human pixel blocks. apparatus. 9. The number-of-people detecting means detects the number of people by using a neural net from the number of pixels obtained from the number-of-people characteristic detecting unit, a representative point, and a type of form.
The image processing device according to any one of 1 to 8. 10. A feature for estimating a foot position in a foot position detecting means from a human pixel block obtained from a human area detecting unit, a representative point, and a thermal image obtained from an image storage unit in a position determination feature amount detecting unit. 10. The image processing apparatus according to claim 7, wherein the foot position detecting means estimates the foot position using a neural network by calculating the amount. 11. The foot position detecting means according to claim 3,
4. A neural net is provided for each posture to be detected, and the output result of the neural net for estimating the foot position for each posture is selected based on the output result of the posture detecting means. The image processing device according to any one of 10. 12. The output result from the foot position detecting means according to claim 3 is determined according to the attitude determination result output from the attitude detecting means and the number of outputs from the neural network for estimating the foot position for each of the attitudes. The image processing apparatus according to any one of claims 3 to 11, wherein calculation is performed using the corrected value, and the foot position information is corrected and output. 13. The posture determination feature quantity detecting unit according to claim 7, wherein from the human pixel block obtained from the human region detecting unit, the representative point, and the thermal image obtained from the image storage unit,
The image according to any one of claims 7 to 12, wherein a feature quantity for posture estimation in the posture detection means according to claim 3 is calculated, and the posture estimation is performed using a neural network in the posture detection means. Processing equipment. 14. The skin temperature determination feature amount detection unit according to claim 7, wherein the human pixel block obtained from the human region detection unit, the representative point, and the thermal image obtained from the image storage unit are used. Calculate the skin temperature estimation feature amount in the skin temperature detection means, further, in the skin temperature detection means, by the block dividing means having a pattern recognition mechanism, from the feature amount and the posture obtained from the posture detection means, The human pixel block is divided into a plurality of parts according to the body part of the human body, the block temperature calculation means calculates an average temperature for each block obtained from the block division means, and the temperature comparison means obtains the average temperature by the block temperature calculation means. The skin temperature is estimated by comparing the temperature of each block and extracting the block corresponding to the skin surface. The image processing device according to any one of 1. 15. The clothing amount determination feature amount detection unit according to claim 7, wherein the human pixel block obtained from the human region detection unit, the representative point, and the thermal image obtained from the image storage unit are used. A feature amount for estimating a clothing amount in the clothing amount detecting means is calculated, and further, the clothing amount detecting means,
15. The structure of the skin temperature detecting means according to claim 14, wherein a block corresponding to an exposed portion of a human body obtained from the temperature comparing means and an area comparing means for calculating an area ratio of a block corresponding to a clothing surface portion are added. 7. The method according to claim 7,
15. The image processing device according to any one of 1 to 14. 16. The clothing amount determination feature amount detection unit according to claim 7, wherein the human pixel block obtained from the human region detection unit, the representative point, and the thermal image obtained from the image storage unit are used. A feature amount for estimating the amount of clothing in the clothing amount detecting means is calculated, and in the clothing amount detecting means, from the feature amount, the posture obtained from the posture detecting means, and the foot position information obtained from the foot position detecting means. 15. The image processing apparatus according to claim 7, wherein the amount of clothing is estimated by using a neural network. 17. The accumulation processing unit according to claim 5, wherein the accumulation storage unit accumulates the representative points of the human pixel block for a predetermined period, and the stationary object determination unit determines the stationary object from the value of the accumulated value of each pixel. 17. The image processing apparatus according to claim 5, further comprising a stationary object pixel position storage unit that stores the determined pixel position of the stationary object. 18. A human image detecting unit for generating a difference image between two continuous thermal images, and a human body image generating unit for generating an image of a moving object from the two current images and the difference image. Means, a frequency distribution creating means for creating a frequency distribution of pixel values of the moving object image, a cutout temperature determining means for determining a cutout temperature of a person from the frequency distribution, and a human detection unit for performing temperature cutout in the cutout temperature range. 18. The image processing apparatus according to claim 2, further comprising: a representative point calculation unit that calculates a representative point of the cut out human pixel block. 19. The human area correction unit according to claim 5, wherein the thermal image and the position of the pixel cut out from the human pixel block obtained from the human area detection unit determine the connection relationship of the pixel to one human. Pixel connection means, isolated point search means for searching for isolated point pixels that are not connected to other pixels, and peripheral pixels for searching around isolated point pixels and erasing isolated points and complementing isolated points and cut-out pixel blocks The image processing apparatus according to claim 5, further comprising a search unit. 20. An environment information extracting unit according to claim 4, wherein a foot temperature detecting unit detects a temperature near the foot position from a thermal image obtained from the image storage unit and foot position information obtained from the foot position detecting unit, An indoor part temperature detecting means for detecting a temperature of each part based on a correspondence relationship between a pixel and a part such as a wall or a floor in a room, and a radiant temperature detecting part for detecting a radiant temperature near a person. The image processing device according to any one of Items 4 to 19. 21. A floor heating detection means for distinguishing a human body from a floor heating according to the size of a human pixel block obtained from the human area detection means in the environment information extraction portion, and a high temperature from a thermal image obtained from the image storage portion. 21. The image processing apparatus according to claim 20, further comprising at least one auxiliary heating detection unit that detects a heat-generating object. 22. The image processing apparatus according to claim 1, further comprising a console section for a user to input environmental information regarding a detection area that cannot be directly detected from a thermal image, and the information can be reduced. The image processing apparatus is characterized in that the information is also input to the image processing unit or the image information detection unit to correct information related to human beings. 23. The image processing apparatus according to claim 1, further comprising: a visible image input unit, a visible image processing unit that processes the visible image from the visible image input unit to extract a feature amount. A visible image information extraction unit that detects human information from the feature amount from the visible image processing unit is added, and further,
An image processing apparatus, wherein the output from the visible image information detection unit is input to the image information detection unit to correct the output of the image information detection unit. 24. The image processing apparatus according to claim 1, further comprising a visible image input unit, a visible image processing unit that processes the visible image from the visible image input unit to extract a feature amount. A visible image information extraction unit that detects human information from the feature amount from the visible image processing unit is added, and further, the output from the image information detection unit is input to the visible image information detection unit to detect the visible image information. An image processing apparatus, characterized in that the output of the unit is corrected or the information related to a person detected by the image information detection unit is presented on a visible image. 25. The visible image information detecting unit extracts information about a human motion from the feature amount about a plurality of visible images obtained from the visible image processing unit. The image processing device according to claim 1. 26. The image processing apparatus according to claim 23, wherein the output of the visible image information extraction unit or the output of the image information extraction unit is input to the image processing apparatus, and is combined with already input information as monitoring information. An image processing apparatus, characterized by further comprising a monitoring information detection unit for outputting a suspicious person approach warning to a user. 27. The process of the visible image processing unit or the visible image information detecting unit is adaptively performed based on human information detected by the image processing unit or the image information detecting unit according to claim 23. An image processing apparatus characterized in that an adaptive learning section for learning is added. 28. The processing of the image processing unit or the image information detection unit is adaptively performed based on human information detected by the visible image processing unit or the visible image information detection unit according to claim 23. An image processing apparatus characterized in that an adaptive learning section for learning is added. 29. An apparatus using a plurality of image processing apparatuses according to any one of claims 1 to 28, wherein an integrated processing unit that integrates information from each image processing apparatus is added, and a person in a detection area is added. Alternatively, an image processing apparatus characterized by detecting environmental information. 30. A control index having the image processing apparatus according to claim 1 for performing air conditioning from a sensor section, a personal information signal obtained from the image processing apparatus, and an environmental information signal. An air conditioner control device, comprising: a control index determination unit that determines the air conditioner and an air conditioning control unit that controls the air conditioning equipment. 31. An activity amount detection unit, wherein the control index determination unit detects an activity amount representative of an individual or a room from the personal information output from the personal information extraction unit of the image processing apparatus; Comfort level detection for detecting a comfort level for each individual or representative from the environmental information obtained from the environmental information extraction unit, the wind direction / air volume information obtained from the air conditioner, and the activity amount obtained from the activity amount detection unit 31. The control device for an air conditioner according to claim 30, further comprising a unit, which outputs a control target value of the air conditioning control unit from the comfort level. 32. A life scene estimation unit for estimating a life scene from at least the activity amount obtained from the activity amount detection unit, the time, and the environmental information output from the environmental information extraction unit is added to the control index determination unit. Estimating transitional life scenes in which comfort level does not serve as a control index and stable life scenes in which the occupant stays in the room for a predetermined period of time and comfort level can be used as a control index, and 32. The control device for an air conditioner according to claim 31, wherein a predetermined control index predetermined for each comfort level and scene is selected. 33. A remote controller capable of reporting a temperature or comfort level, at least a control index calculated by the control index determining unit, and a user's report from the remote controller are input, and a correction value of the control index is output. 33. The air conditioner according to claim 30, further comprising a remote control operation learning unit having a neural network, and learning the environmental condition when the user performs remote control operation with the above configuration. Control device. 34. An occupancy data storage unit that stores occupancy data, which is a time-series detection result of people entering and leaving the room, and an occupancy pattern that classifies the occupancy data into similar patterns. An absent control index determination unit including a classification unit and a base temperature conversion unit configured to determine a base temperature, which is the minimum maintenance temperature of the absence time, according to the usage status of the room obtained from the presence pattern classification unit is added. The control device for an air conditioner according to any one of claims 30 to 33, wherein: 35. A movement amount detection means for calculating the movement speed of an occupant from the personal information obtained from the personal information extraction unit of the image processing apparatus according to claim 1, wherein the activity amount detection unit comprises: , Personal information storage means for storing the personal information obtained from the personal information extraction unit for a predetermined period, and the frequency of the movement amount value in the predetermined period falling within a pre-divided range are counted and ranked ( Hereinafter, referred to as a period movement amount) period movement amount calculation means, stay frequency calculation means for accumulating the foot position information obtained from the personal information extraction means for each predetermined period position, and a period movement amount and stay The control device for an air conditioner according to any one of claims 31 to 34, further comprising activity amount calculation means for calculating an activity amount from a stationary person determination result obtained from the frequency calculation means. 36. A personal information storage means for storing the personal information obtained from the personal information extraction unit of the image processing apparatus according to claim 1 by the activity amount detection unit, for a predetermined period,
An activity amount is calculated from a stay frequency calculation unit that accumulates foot position information obtained from the personal information extraction unit for each predetermined period and performs stationary person determination, and a stationary person determination result obtained from the period movement amount and stay frequency calculation unit. The control device for an air conditioner according to any one of claims 31 to 35, characterized in that the amount of activity is calculated from a frequency of staying at a floor surface position for a predetermined period, the activity amount calculation means being configured to calculate. . 37. The movement amount detecting means according to any one of claims 1 to 29.
The foot position information obtained from two continuous thermal images obtained from the personal information extraction unit of the image processing apparatus according to any one of item 1 and the foot position that can be moved from the maximum movement distance that can be moved in the measurement unit time. A movable determination means for searching for a combination of information, a nearest neighbor searching means for searching the nearest person in the room from the combination information, and determining a combination,
The moving speed calculating means for calculating the moving speed from a determined combination is provided.
6. The air conditioner control device according to any one of 6 above. 38. The sensor section comprises at least a two-dimensional infrared sensor, an optical sensor, a humidity sensor, a temperature sensor for measuring room temperature, and a temperature sensor for measuring outside air temperature. 38. The control device for an air conditioner according to any one of 37. 39. A calendar, a sensor for detecting a person in the sensor unit, an presence / absence determination unit for determining presence / absence of a person in the room, an in-room control index determination unit for determining a control index when the user is in the room, and a claim. Item 34. An air conditioner control device comprising: a control index determination unit, which is the absence control index determination unit according to Item 34, and an air conditioning control unit that controls an air conditioner. 40. The image processing device according to any one of claims 1 to 29, a traffic flow classification means for classifying a human traffic flow in a transportation means for transporting a human, and a temporal traffic flow of a human being predicted. And a traffic flow control device for controlling the traffic flow based on the predicted traffic flow and information on at least the number of people, which is the output of the predicted traffic flow and the image processing device. 41. A monitoring device comprising the image processing device according to claim 1. 42. A nursing device comprising the image processing device according to claim 1. 43. An image processing apparatus according to any one of claims 1 to 29, further comprising a personal identification section for learning a characteristic amount relating to a person obtained from the image processing apparatus and for identifying an individual. Personal identification device. 44. An applied device using the image processing device according to claim 1.