WO2020218409A1

WO2020218409A1 - Thermosensitive imaging device, watching and monitoring system using thermosensitive imaging device, and watching and monitoring method using thermosensitive imaging device

Info

Publication number: WO2020218409A1
Application number: PCT/JP2020/017468
Authority: WO
Inventors: 渡辺　尚志
Original assignee: 株式会社クロスエッジラボ
Priority date: 2019-04-25
Filing date: 2020-04-23
Publication date: 2020-10-29
Also published as: JPWO2020218409A1; JP6944222B2

Abstract

[Problem] To provide an object person watching and monitoring system with high reliability at a wide viewing angle. [Solution] High-reliability watching and monitoring is achieved by determining the position, posture, action, situation, and the like of an object person by analyzing a temperature image outputted from a thermosensitive imaging device having a catadioptric system including a convex mirror disposed outside the axis.

Description

A monitoring monitoring system using a thermal imaging device, a thermal imaging device, and a monitoring monitoring method using a thermal imaging device.

The present invention relates to a thermal imaging device, a monitoring monitoring system using a thermal imaging device, and a monitoring monitoring method using a thermal imaging device.

With the progress of the aging society, the number of elderly people who need long-term care and elderly people living alone is increasing. Appropriate support is required to protect the daily safety and health of such elderly people. When an accident such as a dementia patient, a resident of a long-term care facility, a hospitalized patient, an elderly person living alone, or an accident such as a fall or a fall occurs in a private room, the family or the caregiver does not notice and the response is delayed. In such cases, the symptoms become severe and, in the worst case, fatal events occur.

Various private room monitoring systems have been proposed in order to detect illnesses and accidents in such private rooms at an early stage. The elderly watching system includes a method of detecting the state of the person being watched over by using a sensor and a method of observing the state of the person being watched by an image such as a video camera.

In the method using sensors, a technique has been proposed in which the state of the target person is detected by a plurality of sensors such as a pyroelectric infrared sensor, a pressure sensitive sensor, an optical sensor, and an ultrasonic sensor, and the detected state is notified. (Patent Document 1). On the other hand, as a technique for recognizing the state of a subject using an image of the subject, a technique of relaying the image of the subject to a caregiver has also been proposed (see Patent Document 2).

If the biometric information and behavior of the elderly person to be watched can be grasped by such an elderly person watching and monitoring system, it is expected to be useful not only for ensuring the safety of the elderly but also for health promotion and health care.

Although the number of fatalities in traffic accidents is declining, it still exceeds 3,000 a year. A characteristic of traffic fatalities in Japan is the high rate of accidents while walking or riding a bicycle. In particular, pedestrian and bicycle accidents increase at night. This is because it is harder to find pedestrians and bicycles at night than at noon. In order to solve such a problem, a vehicle equipped with a far-infrared camera (thermography camera) capable of detecting a pedestrian from body temperature even at night has been developed and is already on the market (see Patent Document 3).

Japanese Unexamined Patent Publication No. 2001-327549

Japanese Unexamined Patent Publication No. 2000-105885

Japanese Unexamined Patent Publication No. 60-231193

Among conventional elderly watching systems, single-function sensors such as pressure-sensitive sensors and optical sensors are used to detect specific movements such as getting up and getting out of bed, and when such movements are performed, Generate an alarm. With such a simple sensor, it is not possible to determine the detailed situation of the person to be watched over, so it is difficult to accurately determine whether the situation really requires assistance. For this reason, there is a problem that many unnecessary alarms are generated and the burden on the caregiver is increased.

Also, depending on the sensor settings, the behavior that should be detected was often overlooked. As a method for monitoring the behavior of the subject in more detail, a behavior monitoring system using a video camera has been put into practical use. However, surveillance using a video camera has a large privacy problem and has not been widely used in the living space of the elderly. This is because there are problems such as a large psychological resistance of the person being watched over, a large physical and psychological burden on the caregiver to confirm the image, and a risk of leakage of images and the like. Further, if the image is simply relayed, it is necessary to constantly monitor the image, which causes a problem that the burden on the caregiver is heavy and there is a risk of overlooking.

In order to solve such problems, a system has been proposed that uses an image recognition program to determine and warn the subject's unusual behavior or dangerous state. However, even with such a system, it is difficult to appropriately detect an accident or abnormality of the subject from a complicated image, and there is a problem that an oversight or a false alarm occurs. In addition, after detecting an abnormal state, it is necessary for the caregiver to judge the necessity of assistance by looking at the actual image, and there is also a problem that a privacy problem occurs here.

Furthermore, when watching over using a video camera, a high-resolution image is required to recognize the target person and their behavior in the presence of various objects in the room. The use of high resolution images causes privacy issues. In addition, a computer with a large computing power is required for image recognition for analyzing a high-resolution image with a computer and recognizing the position and behavior of the person being watched over. For this reason, in a watching monitoring system using a video camera indoors, an image is often sent to a host computer outside the private room where image monitoring and image recognition are performed. A high-priced image transmission device and a host computer are required for relaying such images, and at the same time, there is a risk of image leakage.

There is a need for a monitoring monitoring system that can accurately grasp the current situation of the target person while giving consideration to privacy, accurately detect the situation requiring assistance, and issue an alarm. In addition, it is desirable that the watching monitoring system is small and arranged so as not to interfere with the daily activities and assisting movements of the monitoring target person and not to interfere with the daily life of the target person and the caregiver. Furthermore, it is desirable that the system has a simple configuration and is inexpensive so that the cost of nursing care and watching can be suppressed.

Next, the issues of the existing in-vehicle monitoring monitor system will be described. The problem with in-vehicle surveillance systems using existing far-infrared cameras is that the viewing angle is narrow. The viewing angle of an in-vehicle far-infrared camera currently on the market is 40 degrees or less, and it is a system for detecting pedestrians and animals in the distance. This is because the field of view of the conventional far-infrared camera is narrow and it is not possible to realize a wide field of view.

Accidents between automobiles and pedestrians / bicycles often occur in the city, especially when turning left or right, and often caused by oversight of crossing pedestrians. In order to prevent these accidents, it is important to be able to detect pedestrians and bicycles in the vicinity of the vehicle. For that purpose, it is important to be able to monitor the vicinity of the vehicle with a wide field of view.

An object of the present invention is to provide a thermal imaging device, a watching monitoring system, and a watching monitoring method that can detect a target person in a wide field of view with a simple configuration.

In order to solve the above-mentioned problems, in the present invention, monitoring is performed using a far-infrared camera (thermography) capable of photographing in a wide viewing angle. A far-infrared camera is a device that captures a temperature distribution calculated from the energy of far-infrared rays emitted from an object as an image. Since humans have a body temperature, which is higher than the normal environmental temperature, humans can be selectively detected from the body temperature. Therefore, unlike a normal video camera, it is not necessary to acquire a high-resolution image to recognize a human from the image, and a human can be selectively detected from a low-resolution temperature image, so that there are few recognition errors. The computing load on the computer can be reduced. Therefore, processing can be performed by a computer having a small computing power, and the system can be reduced in price and size.

Since a person can be detected directly from the body temperature, a monitoring device using a far-infrared camera has been conventionally proposed. The problem of the monitoring monitoring system using such a far-infrared camera is that the imaging range of the far-infrared camera is narrow. Since the viewing angle of a commercially available far-infrared camera is about 40 to 60 °, it is difficult to monitor and monitor a wide range. For example, as an example of an elderly person watching system, there is "Non-patent document: Takagi et al." Watching elderly people using a temperature sensor ", DEIM2016, P6-1, 2016". In this system, a far-infrared camera was installed on the ceiling on the bed or on the wall surface facing the bed to monitor the behavior on the bed (such as detection of falling from the bed). In such a system, there is a problem that the watching range is limited to the bed.

An example of forward monitoring of an automobile is a night vision system using a far-infrared camera manufactured by Autoliv installed in the BMW 7 series. In this system, the viewing angle is set to 36 degrees during normal driving and 24 degrees during high-speed driving. Therefore, although it was possible to detect pedestrians and animals in the distance, it was not possible to detect pedestrians in the short distance. This is also because the field of view of the conventional far-infrared camera is narrow and a wide field of view cannot be realized.

In order to solve the problem that the field of view of a far-infrared camera is narrow and it is possible to monitor and monitor only a limited area, research is being conducted on scanning a far-infrared camera with a motor to capture a wide range (non-patent documents). : Shimamura, "Understanding the Position and State of People Using Spatial Sensors", Bulletin of Hosei University Graduate School, 5, 2016-03-24). In such a method, since the device includes moving parts, the device is complicated and there is a concern that a failure may occur. In addition, it is necessary to perform a process of stitching images together, and it takes time to acquire an image of the entire area. There was a risk of overlooking.

An example of realizing a monitoring device with a wide viewing angle by combining a far-infrared camera and a curved mirror arranged on the optical axis has also been reported (Non-patent document: Wong Wai Kit et al., "Omnidirectional Thermal".
Imaging Surveillance System Featuring Trespasser and Faint Detection ”,
International Journal of Image Processing, Vol 4 (6), pp.518 (2011).). In this example, a hyperboloid mirror is arranged on the optical axis of the far-infrared camera to realize a wide field of view. This method has a problem that a blind spot is generated in the center of the visual field.

If a far-infrared camera that can simultaneously image the entire indoor area without blind spots can be realized, these problems can be solved and a practical far-infrared camera can be realized. In the present invention, by using a heat-sensitive image pickup device having a reflection-refraction type optical system off the optical axis, a heat-sensitive image pickup device having an imaging range in a wide area without a blind spot is realized, and a monitoring monitoring system using this device is realized. Was realized.

By using a heat-sensitive image pickup device that has an optical system that is off-axis and reflects and refracts, a wide imaging range can be realized without blind spots. Since the thermal image pickup device can selectively detect a human from a low-resolution image, there is an advantage that image recognition becomes easy and recognition accuracy can be improved.

Thermal imaging device Conventional far-infrared camera with wide field of view Explanatory drawing showing the imaging range of the catadioptric system Explanatory drawing showing a cross-sectional shape of a convex mirror Figure comparing spherical mirror and parabolic mirror Overall configuration diagram of the elderly watching system installed in the living room of the elderly facility Top view of the indoor installation layout of the thermal imaging device Indoor installation bird's-eye view of thermal imaging device Two-dimensional temperature image of the room Explanatory drawing showing the required viewing angle of the thermal image pickup device Indoor imaging temperature image of thermal imaging device Indoor captured image of visible light camera Explanatory drawing of initial settings for room layout and target audience Relationship between head size and head position Relationship between the distance between the thermal imaging device and the head and the measured temperature Body movement and sleep depth measurement results on the bed Actual measurement data of chest position by head temperature and respiration Results of frequency analysis of chest position by head temperature and respiration Pictogram display example of posture recognized from temperature image and temperature image Posture and situation of the person to be watched, pictogram display example when an alarm occurs Thermal imaging device with a design different from that of FIG. Flow chart of watching monitoring system Conceptual diagram of a monitoring system for the elderly living alone. Conceptual diagram of a monitoring system for nursing homes and hospitals Daily life analysis data using the monitoring monitoring system Monthly life analysis data using a watching monitoring system Annual life analysis data using a watching monitoring system Monthly walking count analysis data using a monitoring monitoring system Conceptual diagram of watching service and rush service Thermal imaging device using a cylindrical mirror Explanatory drawing showing the horizontal imaging range of a cylindrical mirror Explanatory drawing which shows the imaging range of an in-vehicle thermal image pickup apparatus The figure which shows the imaging range on the road of the in-vehicle thermal image pickup apparatus Temperature image of pedestrians on the road using an in-vehicle thermal imager Thermal imaging device using a saddle-shaped mirror Application example of thermal imaging device to driver and passenger monitors Temperature image inside the car Application example of thermal imaging device for autonomous vehicles for delivery

About the thermal imaging device for watching

The heat-sensitive imaging device 10 having an imaging range of the entire room will be described with reference to FIG. The heat-sensitive image pickup device 10 is a single board computer 5 connected to a far-infrared camera 3 including an image pickup lens 3-1 and a far-infrared array sensor 3-2, a convex mirror 2, a far-infrared camera 3, and a wiring 4. Is integrated by the frame 1. Here, the characteristic is that the optical system of far-infrared light is an off-axis reflection-refraction type optical device that combines an image pickup lens 3-1 that utilizes refraction and a convex mirror 2 that utilizes reflection. is there. Here, the optical axis off-axis optical system means that the optical axis 3-4 of the convex mirror 2 is inclined with respect to the optical axis 3-3 of the far-infrared camera 3. The convex mirror 2 has an axisymmetric three-dimensional shape. It can be seen that by using the reflection of the convex mirror 2, the viewing angle of the far-infrared camera 3 is expanded so that a wider range can be imaged. When the thermal imaging device 10 is installed on a wall surface near the ceiling, it can be seen that a wide field of view including the ceiling region and almost the entire wall surface region can be obtained as shown by the typical light beam 6.

Since the convex mirror 2 has a rotationally symmetric shape, it has a wide field of view even in the depth direction of the paper surface of FIG. As a heat-sensitive imaging device that realizes a wide imaging range by utilizing the reflection of the convex mirror 2, the convex mirror 2 as shown in FIG. 2 is arranged on the far infrared camera 3 with the optical axis aligned, and is far away. An attempt to make the infrared camera 3 omnidirectional (360-degree field of view) has been reported (Non-patent document: Eiichi Narimatsu, "Development of omnidirectional thermograph for finding and tracking people indoors", Proceedings of the 21st Annual Meeting of the Robotics Society of Japan, 2003). In the example of FIG. 2, a hyperboloid mirror is used as the convex mirror 2. In an optical system having such a configuration in which the optical axes of the convex mirror 2 and the far-infrared camera 3 are aligned with each other, there is a problem that a blind spot occurs because the far-infrared camera 3 is reflected in the center of the field of view. There was a problem that it could not be used for watching purposes.

In order to solve this problem, in the present invention, the far-infrared camera 3 is arranged at an angle with respect to the convex mirror 2, and as shown in FIG. 1, a blind spot is provided by using a reflection-refraction type optical system off the optical axis. A wide range of images can be acquired without it. It can be seen that by using the convex mirror 2, the viewing angle is wider than the viewing angle of the far infrared camera 3. Further, the catadioptric system using the convex mirror 2 will be described in detail with reference to FIG. The optical axis 3-3 of the far-infrared camera 3 is tilted with respect to the optical axis 3-4 of the convex mirror 2.

At this time, the effective imaging range of the convex mirror 2 is the light beam 6-1 in the normal direction of the convex mirror 2 passing through the exit pupil center point 3-3 of the imaging lens 3-1 and the exit pupil of the imaging lens 3-1. It is within the range of the light beam 6-2 in the tangential direction of the convex mirror 2 passing through the center point 3-3.

Assuming that the angle formed by the optical axis 3-3 of the far-infrared camera 3 and the optical axis 3-4 of the convex mirror 2 is α and the total viewing angle of the far-infrared camera 3 is θ, the direction of the normal direction of the convex mirror 2 is In order for the ray 6-1 to be included in the field of view, θ / 2 <α
It is necessary to satisfy the condition.

On the other hand, in order for the tangential ray 6-2 of the convex mirror 2 to point upward from the horizontal direction, θ / 2 <.
It is necessary to satisfy the condition of π / 2 -α. This is to prevent a blind spot from being generated on the wall surface facing the wall surface on which the thermal image pickup device 10 is installed when the tangential ray 6-2 is directed downward from the horizontal direction.

In order to satisfy both of the above two conditions, the angle α formed by the optical axis 3-4 of the convex mirror 2 and the optical axis 3-3 of the far-infrared camera 3 is θ / 2 <α <(π-θ) /. It is necessary to satisfy the relationship of 2. The viewing angle of the far-infrared camera 3 may differ between the horizontal direction and the vertical direction, in which case θ corresponds to the smaller viewing angle.

Next, the shape of the convex mirror 2 will be described. A wide viewing angle can be obtained by using the convex mirror 2, but the distortion of the image becomes large. By changing the mirror shape, it is possible to control the distortion characteristics of this image. For example, it is possible to magnify the area of interest in the image and shoot.

It is important to increase the resolution on the bed for watching over the elderly. This is because elderly people spend a lot of time on the bed, and it is important to acquire sleep data in order to understand their health condition. By increasing the magnification of the part where the bed is photographed in the image pickup screen of the thermal image pickup device 10, sleep can be converted into data with high accuracy.

As the cross-sectional shape of the convex mirror 2, a flat ellipse, a circle, an ellipse, a parabola, a hyperbola, or the like can be considered as shown in FIG. The curves shown in the figure are compared with figures in which the curvature at X = 0 is constant. In order to compare the effects of the mirrors, the reflected rays of the spherical mirror and the parabolic mirror are shown in FIG. The straight line shows the spherical mirror, and the broken line shows the reflected light beam of the parabolic mirror. The light rays show the reflection of the light rays from the far-infrared camera at every 2.5 ° viewing angle on the mirror.

Comparing the reflected rays of the spherical mirror and the parabolic mirror, the rays reflected at the lower part of the convex mirror (around X = 0) are almost the same, but between the reflected rays of the parabolic mirror toward the outer periphery of the mirror. It can be seen that the angle of is smaller (the magnification is higher). It can be seen that the relative magnification in the screen can be controlled by changing the shape of the convex mirror.

Optical systems using mirrors have been studied a lot with ordinary cameras for visible light. In recent years, optical material technology and lens processing technology for refracting lenses for visible light and near-infrared light have been rapidly advanced. For this reason, the performance, miniaturization, and low price of lenses with various characteristics have made rapid progress. As for fisheye lenses and ultra-wide-angle lenses that realize a wide field of view, high-performance lenses are available at low prices, and reflective optics are rarely used except for special applications.

On the other hand, far infrared rays require special materials such as germanium, silicon, and chalcogenide, and the lenses are expensive and the market is small, so it is difficult to obtain special lenses such as wide-angle lenses. In the present invention, a thermal imager having a wide field of view by combining a standard far-infrared camera (viewing angle of about 50 °) that can be obtained at a low price and a small size with a metal convex mirror that is easy to process and can be obtained at a low price. 10 is realized.

(Example 1)
As the first embodiment of the monitoring monitoring system using the heat-sensitive image pickup device 10 having the optical axis off-axis reflection / refraction type, elderly people such as nursing homes, elderly housing with services, special elderly nursing homes, and group homes. The elderly watching system installed in the living room of the facility will be explained. The overall configuration of the elderly watching system is shown in FIG. The elderly watching system has two functions: a watching function that monitors the safety of daily life of the elderly, and a healthcare function that uses the living data of the elderly acquired by the system to improve the health of the elderly. ing.

The heat-sensitive imaging device 10 of the present invention is installed in a private room which is a living space for the elderly. When the system watches over the occurrence of an accident such as a fall or a fall, or an abnormality such as a sudden change in physical condition, the occurrence of the abnormality is sent to the mobile terminal 21 of the caregiver or guardian 23, and the caregiver or guardian 23 rushes in. And protect the safety of the elderly. By accumulating daily life data obtained by watching over and analyzing it as big data accumulated on the cloud, we analyze short-term changes in physical condition, long-term health condition, physical strength, and cognitive function of the elderly. However, by giving appropriate advice, long-term health promotion (health care) for the elderly will be realized.

The monitoring monitoring device using the heat-sensitive image pickup device 10 off the optical axis and the reflection / refraction type shown in FIG. 1 will be described in detail below. The far-infrared camera 3 forms an image of far-infrared rays emitted from an object on a two-dimensionally arranged far-infrared array sensor 3-2 by a refracting lens 3-1 for far-infrared light. It is a device that converts the output signal for each pixel into temperature and displays the temperature distribution as an image. The heat-sensitive image pickup device 10 used here includes a far-infrared camera 3 composed of a microbolometer-type 80x60 pixel far-infrared array sensor 3-2, a silicon refraction lens 3-1 and a convex mirror 2. Will be done. As the convex mirror 2, an aluminum-polished mirror having a rotating paraboloid shape was used. The imaging range of the far-infrared camera 3 used is 51x38 ° in terms of viewing angle, and this viewing angle is magnified by the convex mirror 2 for imaging.

The far infrared camera 3 is connected to the single board computer 5 by wiring 4. The single board computer 5 used here is equipped with a 1 GHz single core CPU chip and a Wifi communication chip, and processes the temperature image acquired by the heat-sensitive camera to recognize the behavior of the target elderly person, and it is necessary. In response, information can be transmitted to an external computer or mobile terminal (tablet, smartphone, etc.) by Wifi communication.

Since the single board computer 5 is small and inexpensive, it can be integrated with the far infrared camera 3 and installed in the room of the person to be watched over. The single board computer 5 can directly receive the temperature image output signal of the far infrared camera 3 via the wiring 4 and perform image recognition. Edge computing is the process of performing image processing inside the device using an IoT device that consists of a sensor device and a computer with a built-in communication function as in this example. Edge computing has the advantages of reducing communication load, real-time data processing, and reducing security risks, and is a technology that has been attracting attention in recent years. In the monitoring monitoring system of the present invention, by using the low pixel image of the far infrared camera, the load of image processing is reduced, and edge computing on the single board computer 5 having low computing power is enabled.

The monitoring monitoring system of the present invention is not limited to edge computing, and the arithmetic unit may be an ordinary computer and may be installed in a place (for example, a caregiver's station) away from the place where the thermal imaging device is installed. It is possible.

In this case, the image processing unit, the discrimination unit, the data storage unit, and the output unit are placed on the computer side, and data communication is performed with the far-infrared camera 3 by wireless or wired communication. From the viewpoint of system price reduction and miniaturization, it is efficient to use a single board computer 5 having a communication function, but if a high-performance computer is required for purposes other than watching, edge computing You can use a normal computer without doing this. For example, it is effective to separate the image processing computer from the edge device when performing complicated operations such as automating the input to the long-term care diary based on the monitoring result.

Here, a parabolic aluminum polishing mirror was used as the convex mirror 2. The aluminum polishing mirror was created by pressing an aluminum plate into a parabolic shape and polishing the surface. Here, a metal mirror is used, but the present invention is not limited to this, and a mirror in which a metal film such as silver or aluminum is vapor-deposited on a polymer material (polycarbonate, cycloolefin, etc.) that has been injection-molded into a parabolic shape can also be used. it can. By using a mirror in which metal is vapor-deposited on plastic, the manufacturing cost of the mirror can be reduced. Although a parabolic mirror is used here as the convex mirror 2, a spherical mirror, a conical mirror, or a hyperboloid mirror can also be used.

FIG. 7 shows an installation diagram of the above thermal image pickup device 10 in a room. There is a person to be watched 11 in the room, and a bed 12, a door 13, a shelf 14, a window 15, a desk 17, and a chair 18 are arranged on the floor 16. The thermal image pickup device 10 is arranged near the ceiling of the wall surface closest to the longitudinal direction of the bed 12. FIG. 8 shows a bird's-eye view of the arrangement in the room. The three-dimensional coordinates of the head are written as (X0, X0, Z0), and the head temperature is written as T0.

FIG. 9 shows a temperature image taken in the room shown in FIG. 7. Since it is an image reflected by a curved mirror, it is an image with a large distortion, but the floor 16 and the four walls surrounding the bed 12 are imaged. The four walls are photographed almost up to the ceiling, so you can understand the wide viewing angle. Since the person to be watched 11 has a higher temperature than the surroundings, it can be clearly identified. Despite the low resolution (80x60 pixels) image, the head, body, and legs are clearly separated, from which the posture of the subject 11 can be determined. The head is indicated by a cross mark in the image. Here, the coordinates on the temperature image of the central part of the head are (x0, y0), and the measured temperature of the head is t0.

It should be noted in the temperature image of FIG. 9 that the existence and posture of the subject 11 can be recognized, but the individual identification and detailed movement of the subject 11 cannot be identified, and the risk of privacy invasion is low. When a normal camera is used, a high-resolution image is required to recognize a human with high accuracy, and the face, facial expression, and action of the subject 11 are imaged, so that the psychological resistance of the watching subject is increased. growing. In addition, in high-resolution images, identification of objects placed in the room and clutter in the room are also observed, which also increases psychological resistance. The greatest merit of using the heat-sensitive image pickup device 10 for watching over is that the heat-sensitive image pickup device 10 can image the heat generated by the human body, so that the human body can be detected with high accuracy with a low-resolution image in consideration of privacy.

FIG. 9 shows the upper part of the door 13, the upper part of the window 15, and the main body of the thermal imaging device 10 installed near the ceiling, and it can be seen that the images are taken from the floor 16 to the vicinity of the ceiling without blind spots. Since the entire wall surface including the door 13 and the window 15 is photographed, the position of the person to be watched in the room 11 can be grasped without a blind spot, and it is possible to detect going out from the door 13 to the outside without overlooking.

Here, the reason why the thermal image pickup device 10 is arranged near the ceiling of the wall that is in close contact with the long side side of the bed 12 is to watch the watch target person 11 on the bed 12 with higher accuracy. Priority was given to watching over bed 12 when the subject 11 was watching over nursing care facilities, hospitals, etc., (1) staying on bed 12 for a long time, and (2) falling from bed 12 or getting out of bed. This is because it takes into consideration that there are many accidents such as falls, and (3) grasping the sleep state is effective for grasping the life and health state of the subject 11. The arrangement position of the heat-sensitive image pickup device 10 is not limited to the bed 12, but fixing the arrangement is also advantageous for improving the accuracy of image recognition. For example, if the approximate position of the bed 12 is known in advance on the screen, it is easier to recognize the behavior of the subject 11 such as being in bed, getting out of bed, getting up, etc., as compared with the case of performing image recognition without such information. And it becomes possible to perform with high accuracy

Further, in this embodiment, in order to measure biological information (sleep depth, respiration, heartbeat, etc.) during sleep, the imaging magnification on the bed 12 can be increased as compared with the spherical mirror as shown in FIG. A parabolic mirror is used. The convex mirror 2 is not limited to a parabolic mirror, and an ellipsoidal mirror and a hyperboloidal mirror can also be used. Further, since the spherical mirror has an advantage that it is easily available and low in cost, it is possible to use a spherical mirror depending on the application.

By using the off-axis reflection / refraction type optical system using the parabolic mirror of the present invention, it is possible to observe in detail the object (here, the bed 12) located diagonally below the camera installation position. This is a great advantage over the case of using a refracting wide-angle camera. When the imaging range extends to the upper part of the wall surface with an optical system using a refraction type wide-angle lens, the bed 12 placed diagonally below the camera appears small in the entire screen, so the situation of the subject 11 on the bed 12 can be seen. It becomes difficult to observe in detail.

Here, the viewing angle required for the thermal imaging device 10 for watching will be described. The watching imager is required to have the ability to watch the subject 11 in the entire area of the room without blind spots. Going out of the room of the watching subject 11 may lead to a serious accident (for example, wandering of a dementia patient) depending on the medical condition and condition of the subject 11, and reliable detection without overlooking is required.

FIG. 10 shows the positional relationship between the subject 11 and the thermal imaging device 10 when going out of the room. Here, it is assumed that the height of the subject 11 is 170 cm, the ceiling height is 240 cm, the room size is 8 tatami mats (13 square meters), and the installation height of the thermal image pickup device 10 is 230 cm. At this time, if there is an imaging range up to a height of 100 cm from the floor surface at the position of the door 13, the buttocks of the subject 11 are caught in the vicinity of the door 13 at the edge of the room, and the behavior of the subject 11 disappearing from the door 13. Can be captured. It can be seen that the risk of overlooking is almost eliminated if an image can be taken up to 150 cm from the floor surface. Even if the tall subject 11 is taken into consideration, if the height from the floor surface to 190 cm is imaged, the head is captured and there is no possibility of overlooking. Therefore, the viewing angle of the wall surface of the thermal imaging device 10 needs to be at least 100 cm or more from under the floor, and it is possible to image an area of preferably 150 cm or more from the floor surface, and particularly preferably 190 cm or more from the floor surface. It is required to be. Under the above conditions, the total viewing angle (2β) required for the thermal imaging device 10 is 140 ° when the imaging range is 100 cm or more from the floor surface, 161 ° at 150 cm, and 167 ° at 190 cm.

There is no inexpensive far-infrared camera with such a wide viewing angle. This is the reason why a new catadioptric system for far infrared rays is adopted in the present invention.

FIG. 11 shows a comparison of the imaging temperature image of the thermal imaging device 10, and FIG. 12 shows a comparison of images captured by a normal video camera. In both cases, the watching subject 11 photographed the scene exiting from the door 13. Here, the optical system of the video camera is not a wide-angle refraction lens normally used, but an optical device corresponding to visible light of the same off-axis reflection / refraction type as the heat-sensitive image pickup device 10 is prototyped and photographed. 11 and 12 are images of the same scene captured so as to have substantially the same field of view.

In both images, it can be seen that the entire area of the room centered on the bed 12 was taken without blind spots. The resolution is 80x60 pixels for the temperature image and 1920x1080 (HD) for the video camera. Although the video camera image shown in FIG. 12 is taken at a higher resolution, it is not easy to visually recognize a person in the image because various objects having different brightness are shown.

On the other hand, in the temperature image by the thermal image pickup device 10 shown in FIG. 11, it can be seen that the human body can be easily detected because the heat generation of the human body can be captured. The superiority of watching over can be understood from the comparison of these two images. It is difficult for a normal visible camera to recognize the target person 11 by image processing and grasp its behavior. On the other hand, the subject 11 can be easily monitored from the body temperature from the temperature image of FIG. 11 by the thermal image pickup device 10.

About behavior recognition technology from temperature images

Hereinafter, an image recognition technique for detecting the monitoring target person 11 from the temperature image and recognizing the behavior will be described. In image recognition, it is known that it is effective to limit the objects to be imaged in order to improve the recognition accuracy. The image recognition accuracy can be improved by limiting the objects appearing in the image to be recognized and by limiting the imaging range, lighting conditions, and imaging conditions.

By fixing the image pickup device and imposing restrictions on the image pickup conditions, image recognition becomes possible with high accuracy. In the present invention, the dedicated heat-sensitive image pickup device 10 whose imaging range is a wide area without blind spots is installed in a limited location near the ceiling of the wall near the bed 12, and image processing is performed on the output image. High recognition accuracy was achieved by optimizing.

A wide viewing angle can be obtained by using a convex mirror, but there is a problem that the image has a large distortion. By performing image processing according to the characteristics of this image distortion, the influence of the image distortion can be reduced and the recognition accuracy can be improved. A heat-sensitive image pickup device with a special optical system called a dedicated far-infrared optical axis off-axis reflection / refraction optical system with a wide field of view is placed in a limited installation location to perform image processing according to the characteristics of the image to be captured. By doing so, a high-performance monitoring and monitoring system was realized.

The image recognition method will be described below. By restricting the image pickup device and its arrangement method as described above (input image constraint), it becomes easy to accurately grasp the behavior of the target person. Furthermore, by utilizing the information about the imaging target to be grasped in advance (prior information), it becomes easier to grasp the behavior. At the time of startup after the installation of the thermal imaging device, information on the layout of the room and information on the person to be watched are acquired in advance, and image recognition is performed using that information to accurately monitor the behavior of the person to be watched. It made it possible to grasp.

An example of inputting prior information in the initial setting after installation of the thermal image pickup device will be described with reference to FIG. The setting is performed by an external computer using the communication function of the single board computer 5 of the thermal image pickup device 10. First, enter the room area. On the selection screen, enter the room size from 4.5 tatami mats (7.3 square meters), 6 tatami mats (9.7 square meters), and 8 tatami mats (13 square meters). If there is not enough space for the selection screen, keyboard input is performed in square meters. The size of the room may be an approximate area.

Next, enter the room shape based on the bed. If the room is rectangular, enter a horizontal rectangle if the longitudinal direction of the bed matches the longitudinal direction of the room, and enter a vertical rectangle if the directions are orthogonal. Next, the bed position is input. Enter the position of the bed on the wall surface close to the bed using A to C. Then, the door position is selected from 1 to 9 and input. Further, the installation height of the thermal image pickup device 10 and the height of the bed are input. Finally, input information about the person to be watched over. Here, the gender, age, height, and weight of the subject are input.

With the above initial settings, the positional relationship between the thermal imaging device 10 in the room of FIG. 7, the bed 12 and the door 13 was monitored and input to the monitoring system. By grasping the size and shape of the room, it is possible to recognize the position and posture of the watching target person 11 with high accuracy. Further, by grasping the position of the door 13, the recognition accuracy of the entrance / exit room can be improved.

After the initial setting is completed, the position detection algorithm of the target person 11 is started. The thermal image pickup device 10 is set to output 5 temperature images per second (5 FPS). The image file is set to output 16-bit 80x60 pixel data in PGM format. The image recognition algorithm will be described with reference to FIG. The output temperature image 19 is shown in the upper left of FIG.

First, the head is detected from the temperature image 19. Usually, the head has the highest temperature in the temperature image 19, and there is little problem even if the highest temperature portion in the temperature image 19 is recognized as the head position. However, for example, when a pot or hot food or beverage is placed indoors, these hot objects may be mistakenly recognized as the head. To avoid this, the head is recognized from the size and temperature of the detected high temperature region. The temperature of the head is measured in the range of 28-36 ° C. depending on the distance between the head and the thermal imaging device 10 and the position of the head in the field of view to be imaged.

As the distance between the subject 11 and the thermal imaging device 10 gets closer, the size of the head to be imaged increases and the temperature also rises. The size and temperature of the head also change depending on the position in the field of view of the temperature image 19. There is a relationship between the position in the field of view, the size of the head, and the temperature as described later, and by recognizing the head using this relationship, it is possible to detect the head with high accuracy. The surface temperature of the head is usually measured at 28-36 ° C., and the larger the size of the head (that is, the closer the head is to the thermal imaging device 10), the higher the temperature. By recognizing the head using this relationship, it has become possible to recognize the head even when other high-temperature objects are included in the screen.

The human head is detected from the temperature image 19, and the coordinate position (X0, Y0, Z0) of the head in real space is estimated using the coordinate position (x0, y0) on the image and the measured temperature t0. (Fig. 14). The spatial coordinate position of the head is estimated using the size of the head and the temperature of the head. First, the method of obtaining the head coordinates from the size of the head will be described. A threshold value is set from the temperature image 19 to determine the region of the head, and the semi-major axis radius (a) and the semi-minor axis radius (b) are obtained by elliptical approximation. From the head area S (= πab), use the average radius r = sqrt (S / π) of the head. The semimajor axis of the head of an adult male is 24 cm, the minor axis dimension is 16 cm, and the average height is 170 cm. Since the heat-sensitive image pickup device 10 used in the present invention has a large image distortion, a spherical plastic container having a radius of 10 cm containing hot water (35 ° C.) is placed on the floor as a simulated head for data acquisition, and a screen is displayed. Data on the top position and the size of the captured image were acquired.

A method of obtaining the three-dimensional spatial position of the head from the dimensions of the head on the screen will be described with reference to FIG. In FIG. 14, the position is shown on the xz plane for simplicity, but the same calculation can be performed on the xyz space. From the captured temperature image 19, the center position x0 of the head and the area S0 can be obtained. From the position x1 and area S1 of the sphere with a radius of 10 cm measured on the floor surface and the area S0 actually imaged, the actual head position x0 can be obtained as x0 = sqrt (S1 / S0). Further, the height z0 of the head is obtained as z0 = (1-sqrt (S1 / S0)) * H, where H is the installation height of the thermal imaging device 10.

Next, the method of calculating the coordinate position from the head temperature will be described. In the thermal imaging device 10, the measured temperature is affected by the distance. This is mainly because the area occupied by the target object in the pixels becomes smaller as the distance increases, and it becomes affected by the ambient temperature of the measurement object. This temperature change also changes depending on the position of the temperature image 19.

In FIG. 15, the measured temperature of a spherical plastic container having a radius of 10 cm containing hot water (35 ° C.) at the positions of the center of view (A) and the edge of the field of view (B) is set to the distance between the heat-sensitive image pickup device 10 and the plastic container. It was plotted against it. The distance between the thermal imaging device 10 and the head can be estimated from the pixel position and the measured temperature of the head. If the distance is known, the head position (X0, Y0, Z0) in the space can be obtained from the relationship shown in FIG. As described above, the position of the head in the space can be obtained from both the head size data and the head temperature data. By combining these two measured values, the head position can be measured with high accuracy.

If the position of the head of the subject 11 is known, it becomes possible to estimate the posture and behavior of the subject 11. For example, assuming that the height of the bed 12 is 40 cm, it can be estimated that the subject 11 is in the recumbent position if the head is on the bed 12 and the height of the head is within the range of 40 to 70 cm. If the head is on the bed 12 and the height is within the range of 80-140 cm, it can be estimated that the subject 11 is in the long sitting position. If the head is near the edge of the bed and the height is within the range of 80-140 cm, it can be estimated that the subject is in the sitting position. Since the sitting position is regarded as a preparatory action for standing, if there is a risk in the subject 11 standing up (in the case of a patient who is likely to fall and the fall leads to a serious accident), before standing up. It is possible to recognize the sitting position, which is a preparatory action, and issue a warning. On the other hand, if the height is 140 cm or more in a place other than the bed 12, it is estimated to be standing. When the head position is moving in the standing position, it can be recognized as a walking state.

Furthermore, the walking speed and walking distance can be calculated from the relationship between time and movement amount. The exercise data acquired in this way can be utilized for the health care of the subject 11. In addition, if the head height is 50 cm or less outside the bed 12, there is a possibility of falling. When the height of the head suddenly drops from the standing position outside the bed 12, it is determined that the person has fallen and an alarm can be issued. Further, when the head height is 50 cm or less outside the bed 12 from the lying position, the long sitting position, and the end sitting position on the bed 12, it is determined that the person has fallen from the bed 12 and an alarm can be issued.

By grasping the indoor layout and the spatial position of the head in this way, it becomes possible to recognize the state (posture and movement) of the subject 11 with high accuracy. If the posture and exercise of the subject 11 in daily life can be recognized, it becomes possible to detect a dangerous state such as an accident of the subject 11 and grasp a long-term change in the health condition. By using the heat-sensitive image pickup device 10 of the present invention, the human body can be clearly identified as a high temperature portion. The bed 12 is shown in the center of the visual field, and the posture of the subject 11 such as lying down, sitting on the edge, standing, walking, and going out can be identified from the positional relationship between the bed 12 and the human body. Although the posture can be recognized with some accuracy only by the position of the head displayed on the image, it is possible to determine the posture with higher accuracy by knowing the coordinates of the head in the space.

Furthermore, it is possible to determine the awakening / sleeping state and the light / deep sleep during sleep from the amount of body movement on the bed 12. It is known that the frequency of appearance of body movements during sleep decreases as the sleep stage becomes deeper. If the number of body movements exceeding the threshold value that occurs in 15 minutes during sleep is defined as the body movement density, the body movements occur continuously during REM sleep or light sleep, and the body movement density increases. However, there are individual differences in this body movement density, and it is difficult to estimate the sleep depth using a uniform threshold value. Therefore, the average value of the body movement density during one week of sleep of the subject 11 is calculated, and light sleep occurs when the body movement density exceeds the average value of the body movement density, and deep when the body movement density is lower than the average value. Determined to be sleep.

FIG. 16 shows the amount of body movement on the bed 12 measured using the heat-sensitive image pickup device 10 of the present invention and the sleep depth calculated based on the amount of body movement. It can be seen that the sleep depth can be measured based on the body movement data. Since it is considered that the change in the sleep state of the subject 11 is linked to the health state, it is possible to grasp the change in the health state by using such data and acquiring and analyzing it over a long period of time, and appropriate sleep. It is expected that providing guidance and lifestyle guidance will lead to long-term health maintenance. Long-term sleep irregularities, increased nocturnal arousal, and increased total sleep time are considered signs of cognitive decline. If mild cognitive impairment (MCI) can be detected at an early stage from changes in sleep habits, it is considered to be effective in preventing dementia.

Furthermore, it becomes possible to monitor heartbeat and respiration from changes in head temperature. FIG. 17 shows data obtained by measuring the temperature of the face measured by the heat-sensitive image pickup device 10 and the amount of movement of the chest by respiration measured by the laser positioning meter in a resting state on the bed 12 side by side with the time axis aligned. It was. Both devices were set to acquire data at 0.1 second intervals to acquire data. From the vertical movement of the chest due to breathing, breathing with a cycle of about 4 seconds can be clearly confirmed. On the other hand, the facial temperature is noisy, and biological signals such as respiration and heartbeat cannot be confirmed. The frequency analysis result of the data (512 data) for 51.2 seconds is shown in FIG. From this result, a peak of 2.7 Hz (cycle 3.7 seconds) caused by respiration and a peak of 1.2 Hz (72 BPM) caused by heartbeat can be clearly observed. In this way, it is possible to acquire biological information by frequency analysis of data for a long time. From the analysis results, it was found that it is necessary to analyze the data for a time of 30 seconds or more for respiration and 50 seconds or more for heartbeat. Furthermore, respiration could be measured from the frequency analysis of minute head body movement data. Therefore, by analyzing the change in head temperature during sleep and the amount of head body movement in combination, it has become possible to perform respiratory measurement with higher accuracy.

Compared to the respiratory signal included in the facial temperature data, the heartbeat signal has a high noise ratio and the measurement becomes unstable. In addition, the data shown in the figure was acquired in a stationary state without body movement, so analysis was performed using the data, but it was difficult to measure respiration and heart rate when there was body movement. In principle, it is considered possible to extract biological signals from data accompanied by body movement by body movement correction, but at present, it is difficult to accurately extract biological signals from data containing large noise of body movement. It was. For this reason, in this system, we decided to acquire respiration and heart rate data from sleep data with small body movements.

If a large amount of body movement occurred even during sleep, the data at that time was excluded and not used. It also excludes data for people lying down or wearing a futon on their head. Respiratory rate or heart rate data in a stable state can be compared with the average respiratory rate / heart rate during sleep, and an alarm can be issued if there is an abnormal change in respiratory rate / heart rate. ..

As described above, by recognizing and analyzing the temperature image output from the heat-sensitive image pickup device 10, the position, posture, behavior, and emergency situation such as a fall, sleep state, and heart rate of the person to be watched 11 in the room. , Respiratory rate, etc. can now be detected.

Next, the output related to the current state of the detected target person 11 will be described. Under normal watching conditions, the temperature image shown in the upper part of FIG. 19 is not recorded and output, and the system is set so that the pictogram shown in the lower part of FIG. 19 displays the current state of the subject 11 in real time. Has been done. This is to give priority to the protection of the privacy of the subject 11. However, it is possible to record and output temperature images by changing the system settings depending on the condition of the target person 11, the nursing care system in the case of a nursing care facility, and the surrounding support system in the case of elderly people living alone. It is a system design.

Fig. 20 summarizes the displayed posture and situation detection and the types of alarms. As the posture, lying position, long sitting position, end sitting position, standing position, walking, and going out are determined. The timing of alarm generation varies depending on the medical condition and situation of the subject. The timing of issuing this alarm is also set when the system is started. In the case of a patient who has a high risk of falling by standing up, an alarm is generated in the sitting position, which is a preparatory movement for standing up. When walking is risky, an alarm is issued while standing, and in the case of patients who are restricted from going out, an alarm is issued when they approach the door or when they go out. It also issues an alarm when a fall or fall is suspected due to changes in posture over time. An alarm can be issued even if there is no or very little movement for a long time. Even if the current body temperature, heart rate, and respiratory rate are significantly different from the body temperature, heart rate, and respiratory rate in normal life, an alarm is issued along with information on body temperature, heart rate, and respiratory rate. In this way, by analyzing the temperature image, it is possible to grasp various postures, exercises, and health conditions of the subject and to detect abnormalities.

(Example 2)
As a second embodiment of a monitoring monitoring system using a heat-sensitive image pickup device 10 having an optical axis off-axis reflection / refraction type, an example in which deep learning (deep learning or machine learning) is used to detect a posture and a situation. explain. In this example as well, the target application is an elderly watching system installed in the living room of an elderly facility.

In the first embodiment, a method of using a rule-based algorithm to detect the posture and situation of the subject was explained. It is also possible to use deep learning to detect posture and situation. Hereinafter, a method of performing learning using a temperature image output from the thermal image pickup device 10 by deep learning and determining the posture will be described.

FIG. 21 shows a thermal imaging device 10 having a design different from that of FIG. A parabolic mirror 2-1 was used as the convex mirror. By making the distance between the parabolic mirror 2-1 and the far-infrared camera 3 close to each other, the device is downsized while maintaining the viewing angle. In the example of FIG. 21, the same single board computer 5 as the device shown in FIG. 1 was used, but the curved mirror was downsized and designed to be vertical to reduce the size. Since the processing capacity of the single board computer 5 alone was insufficient, image recognition was performed by connecting Intel's Movidius to a single board computer as an accelerator 5-1 for deep learning.

If a small computer in which an accelerator 5-1 for deep learning and a single board computer 5 are integrated is adopted, the size of the entire device can be reduced to about 3x3x3 cm. Further, in the design of FIG. 21, a far infrared window 7 is used. The material of the far-infrared window 7 is high-density polyethylene having a high transmittance of far-infrared rays. Since the high-density polyethylene is white, it is designed so that the far-infrared camera 3 inside cannot be seen. Making the monitoring monitoring device smaller and less noticeable is also effective in reducing the psychological resistance of being monitored by the installed thermal imaging device 10.

For deep learning, Google's framework Tensor Flow was used, and the learning model was MobileNet. Learning was performed on the cloud, the created learning model was installed on the single board computer 5 of the heat-sensitive imaging device 10, and image recognition was performed on the single board computer 5 (edge computing). Since the CPU of the single board computer 5 was insufficient in processing power and could not recognize the image of 5 FPS, the AI accelerator 5-1 was connected to the single board computer 5 to make a judgment.

Using deep learning, 7 postures (lying position, long sitting position, end sitting position, standing position, sitting position, falling, going out) were determined among the postures shown in FIG. First, the temperature images of the above 7 postures are acquired. The temperature image acquisition was performed in the room shown in FIG. 200 images in each posture were continuously acquired at 1 per second in order from the recumbent position. Images were acquired while changing the position on the bed 12, the orientation of the body, and the position of the futon little by little while lying in the lying position for 200 seconds. Images of 7 postures were acquired in the same manner by changing the postures in sequence, and 10 sets of images were acquired with this as one set.

The reason why the images were not acquired continuously in the same posture is to avoid being affected by unintended image changes due to changes over time. As a result, the temperature of 2000 sheets in each posture, 14000 sheets in total, was acquired. After attaching label information indicating the posture to all the images, 1000 images of each posture (7000 in total) are used as teacher data images, and a model is created on a notebook PC using the machine learning software library TensorFlow. went.

The model accuracy was confirmed using the remaining 7,000 images that were not used for learning, and 97% accuracy was confirmed. Edge computing has made it possible to determine the posture of the subject in real time from the output temperature image of the thermal image pickup device 10 using machine learning.

Even when machine learning is used, the posture change is determined by the same method as in the first embodiment. When the posture determination is standing, it is determined whether it is standing (resting) or walking by comparing it with the previous image. In addition, a comparison is made with the previous posture, and an alarm is issued when a change occurs from a standing position to a fall, a sitting position to a fall, or a recumbent position to a fall. Even when the person changes from the sitting position to the standing position or goes out, an alarm is set according to the preset alarm setting of the target person.

Regarding machine learning, it is known that the quantity and quality of learning data have a strong influence on recognition accuracy. Two approaches can be considered to improve the recognition accuracy. There are two methods, one is to accumulate the amount of data and improve the accuracy while increasing the number of subjects, and the other is to create a learning model for each subject. When a model is created for each individual, the installation location and target person are fixed, so recognition accuracy can be improved even with a relatively small amount of data. Even when this method is used, it is possible to continuously improve the accuracy by acquiring data and recreating the model while continuing to use it. In this case, it is necessary to acquire data for model creation when the system is installed, but since it is possible to create a model for each individual, highly accurate recognition becomes possible.

(Example 3)
About watching surveillance system including video camera

Next, as a third embodiment of a watching monitoring system using a heat-sensitive image pickup device having an optical axis off-axis reflection / refraction type, a watching monitoring system including a video camera having sensitivity to visible light and near-infrared light. Will be described. Even systems that include video cameras do not shoot video under normal conditions to ensure privacy. When an abnormality is detected from the temperature image of the thermal image pickup device and an alarm is generated, the video camera is activated and shooting is started. When the room is dark, such as at night when an alarm is generated, the near-infrared light source installed in the video camera unit is automatically turned on, and a clear image can be captured. The image of the video camera is transmitted to the information terminal when there is a request from the external information terminal. The caregiver (or family member) can use this image to get a clearer picture of the situation. Whether or not the system includes a video camera is selected according to the wishes of the person being watched over and their parents, caregivers, family members, and facility managers.

FIG. 22 shows a flowchart of control of the monitoring monitoring system. The example of FIG. 22 shows a system that includes visible and near infrared video cameras 32. While the monitoring monitoring system is in operation, temperature images are sent from the thermal image pickup device 10 to the single board computer 5 at regular time intervals. The single board computer 5 has functions as an image processing unit 33, a determination unit 34, a data storage unit 35, and an output unit 36. The image processing unit 33 analyzes the two-dimensional infrared radiant energy image captured by the heat-sensitive image pickup device 10, and detects the position of the subject on the two-dimensional image and the infrared radiant energy contained in the image. The determination unit 34 determines the condition of the subject based on the position on the two-dimensional image of the subject and the infrared radiant energy detected by the image processing unit 33. The data storage unit 35 records the status of the target person determined by the determination unit 34. The output unit 36 outputs information indicating the status of the target person obtained by the determination unit 34.
More specifically, the temperature image (input image) from the heat-sensitive imaging device 10 is sent to the image processing unit 33, and the image processing unit 33 calculates the position of the head in the three-dimensional space. The amount of movement is calculated from the comparison between the head position data of the previous frame and the current head position data. The previous frame referred to here may be a plurality of frames including the immediately preceding frame.

Furthermore, the current posture / behavior of the subject is estimated from the position of the head. The determination unit 34 compares the current position and posture with the position and posture of the front frame, and if there is a sudden change, determines whether or not an abnormality such as a fall or a fall has occurred based on the content of the change. In addition, the determination unit 34 refers to the data regarding the average relationship between the position and the body temperature for each hour stored in the data storage unit 35, compares this data with the data of the current position and the body temperature, and causes abnormal hyperthermia or It is judged to be abnormal when the body temperature is low. The acquired data is stored in the data storage unit 35. Further, the determination unit 34 determines whether or not there is an abnormality in the heartbeat or the respiration frequency. Data of temperature image, head position information, movement amount, posture / behavior estimation value, sleep state, respiration, and heart rate are stored in the data storage unit 35.

When an abnormality is detected by the determination unit 34, an alarm is transmitted from the output unit 36 to an information terminal such as a tablet or smartphone owned by the caregiver or a computer in the office by wireless communication. At the same time as the alarm, the operation of the video camera 32 is started, and the video image is stored in the data storage unit 35. The information to be transmitted is the time of the abnormality occurrence, the content of the occurrence abnormality, the current visible and near infrared images, the current temperature image, and the temperature image for several seconds before and after the abnormality occurrence.

The caregiver who confirmed the alarm confirms the visible and near infrared images and temperature images displayed on the screen, and the temperature images before and after the occurrence of the abnormality, and confirms and assists as necessary. At this time, if the system includes a call function, the mobile terminal may make a call through a speaker to check the status of the target person. Since the caregiver can check the situation of the target person on the spot, it is possible to provide prompt and lean assistance, improve the quality of care, and reduce the burden on the caregiver.

In the above system operation, video recording is started when an abnormality occurs. However, the user may request a clear image when an abnormality occurs. In order to meet such demands, instead of starting video shooting after detecting an abnormality, video shooting is always performed, and video images are stored in the data storage unit 35 for a certain period of time (for example, 30 minutes). , The image data that has passed it is deleted, and when an abnormality occurs, the accumulated image data is not deleted and remains. The video camera is always operated, and when an abnormality is detected, the image before and after the abnormality can be saved by saving the data, and the video image at the time of the abnormality can be confirmed from the mobile terminal. .. In this case as well, the video image is not saved in a normal state in consideration of privacy.

(Example 4)
As a fourth embodiment of the monitoring monitoring system using the thermal imaging device of the present invention, a monitoring monitoring system for the elderly living alone will be described with reference to FIG. 23. A thermal imaging device 10 is installed in the bedroom of the elderly person 11 who lives alone, who is the person to be watched over. When the watching monitoring system detects an abnormality, a mobile terminal or computer terminal is wirelessly communicated to a guardian 23 such as a family member, a close relative, a watching service provider, or a person in charge of a local government registered as a guardian 23 in advance. An alarm is sent to the terminal 21 (not shown). The guardian 23 confirms the terminal display content 22 (warning and temperature image data before and after the abnormality detection) displayed on the terminal 21, and if necessary, rushes, requests an ambulance, and confirms with the residents in the vicinity of the target person. Necessary measures such as request and confirmation by telephone can be taken.

Not only the images before and after the occurrence of the abnormality from the terminal 21 side, but also the current temperature image can be watched and sent from the monitoring system at the request from the terminal 21 side and confirmed on the terminal 21. From the viewpoint of privacy protection, the image transmission is currently limited to the time when an abnormality occurs, and the image request from the terminal 21 is not possible in daily life. At the time of this abnormality confirmation, if the monitoring monitoring system is a system including a video camera, it is possible to confirm the image of the current video camera by selecting the transmission of the video camera image from the terminal 21. .. The monitoring monitoring system can also include devices with remote communication functions such as microphones and speakers. When an abnormality occurs, the guardian 23 can confirm the image on a terminal 21 such as a mobile terminal or a computer terminal, and call the watching target person 11 using the remote call function as necessary. If the call is properly answered, measures such as rushing to the living room will not be necessary. In this way, if the safety status of the subject 11 can be confirmed using the terminal 21, the burden on the caregiver and the family can be reduced.

(Example 5)
As a fifth embodiment, a monitoring monitoring system in a nursing care facility, a hospital, or the like will be described. The watching monitoring system has not only a watching function but also a healthcare function. As shown in FIG. 24, the thermal image pickup device 10 installed in each room is wirelessly connected to the host computer 20 arranged in the facility, and the data in each room is collected by the host computer 20. In this case as well, the analysis of the temperature image of the thermal image pickup device 10 is performed not by the host computer 20 but by the single board computer 5 built in the thermal image pickup device 10. Under normal conditions, the image is not transferred to the host computer 20, and the image data analyzed by the single board computer 5 is discarded as it is if no abnormality is detected, and is stored in the single board computer 5 and in the host computer 20. Not done. In the normal state, the single board computer 5 sends the position data, the posture data, and the data related to the amount of exercise of the monitoring target person 11 to the host computer 20, and the host computer 20 analyzes the life data (life log) of each target person 11. It is put together.

When an abnormality occurs, the single board computer 5 sends an alarm and images before and after the abnormality occurs. These alarm data are sent to the terminal 21 owned by the guardian 23, and after confirming the alarm content, request an image of the current state as necessary and confirm it on the terminal 21. The guardian 23 confirms the call from the terminal 21 by an intercom or the like or by visiting the room.

In the daily state, the terminal 21 of the person in charge of assistance displays a pictogram showing the state of the target person as shown in FIG. 24 in a form that can list each room. The caregiver can confirm the condition of the care recipient on the terminal 21 and decide the procedure of the care work. For example, morning assistance work (breakfast, brushing teeth, changing clothes, etc.) can be carried out in order from the person who is awake, and the person being assisted during sleep can be put off so that the assistance work can be carried out efficiently. Furthermore, it becomes possible to wake up (alarm) at a timing when sleep is light. In this way, by providing assistance in accordance with the life rhythm of the person being assisted, it is possible to improve the efficiency of the assistance work and reduce the burden on the person being assisted. By performing various assistance tasks (for example, medication and changing clothes) in accordance with the life rhythm of the care recipient in addition to getting up, it is possible to reduce the burden on both the caregiver and the care recipient.

In the above system, the temperature image taken in a normal condition is set to be deleted after the judgment without being saved. In long-term care facilities, there is a desire to save images without erasing them in order to grasp the care status and the situation at the time of an accident or the like. In such a case, it is possible to constantly save the temperature image in the host computer 20 by changing the system setting. Since the temperature image has a lower resolution and a smaller data capacity than the commonly used visible light image, it is less restricted by the capacity of wireless communication, and images of a plurality of rooms can be transmitted and received at the same time and stored in the host computer 20. Is possible. Here, the host computer 20 is installed in the facility, but the role of the host computer 20 can be replaced by cloud computing. Since image recognition is performed by the single board computer 5 built into the monitoring monitoring system placed in each room, it is not necessary to send image data to the cloud in real time, so there are issues with communication capacity and communication delay. Does not occur. By eliminating the host computer 20 by cloud computing, the cost of the system can be reduced.

The monitoring monitoring system of the present invention can be useful not only for the monitoring function for the safety of the care recipient but also for the health care of the care recipient through accurate analysis of the daily life of the care recipient. In facilities such as long-term care facilities and hospitals, care recipients spend most of their lives in private rooms or hospital rooms, so the monitoring and monitoring system of the present invention makes it possible to obtain accurate records of daily life. A wristwatch-type or clothing-type wearable sensor may be used for the purpose of monitoring such living conditions, but it is continued because there is resistance to wearing it all the time and the care recipient removes the sensor. There was a problem that it was difficult to monitor. In addition, wearable sensors need to be charged, replaced with batteries, and collected data at high frequency, which increases the burden on caregivers and has not been widely used in long-term care facilities.

In the monitoring monitoring system of the present invention, life can be constantly monitored without being restrained in daily life, and data is automatically sent to the host computer 20, so that both the caregiver and the care recipient bear the burden. It is possible to acquire and analyze living data without any. FIG. 25 shows an example of life analysis. FIG. 25 shows the life pattern during the day. FIG. 26 shows the daily life record of the month in which the data of FIG. 25 is summarized for one month. FIG. 27 shows a monthly life record. FIG. 28 shows the indoor walking distance per day. It can be seen that the amount of exercise decreases from the first half to the second half of the month. From FIG. 26, there is a tendency that the activity decreased in the latter half of the month and the time spent in bed increased.

From the analysis of long-term annual data, it is possible to grasp long-term changes in life. The relationship between sleep state and the progression of dementia and the relationship between daily activity and dementia have been recognized, and it is important to grasp the constant sleep state and motor state for recognition of medical conditions and lifestyle guidance. Similar data analysis is possible for heart rate data and respiration data during sleep. Vital data is useful not only for grasping short-term changes in physical condition and medical condition, but also for health management of long-term care recipients by acquiring and analyzing long-term trends.

By using the monitoring monitoring system of the present invention in this way, it is possible to analyze not only the monitoring for the safety of the care recipient but also the data for the long-term health care of the care recipient and use it for health promotion. become. Furthermore, by acquiring such daily life data (life log) and analyzing the data together with other health data (big data analysis), it is possible to acquire data related to lifestyles for maintaining and promoting health. And analysis becomes possible.

(Example 6)
As a sixth embodiment of the present invention, an example of a watching service will be described. The issue of the monitoring system for safety is the accuracy of abnormality detection. If there are many false positives, the caregiver needs to confirm the safety each time an alarm is issued, which increases the load. The system of the present invention makes it possible to check the situation on a mobile terminal without going to the room and reduce the load, but when there are many false positives, the business is interrupted each time and the alarm is always paid attention. The psychological burden is great. In particular, in a system in which an alarm reaches the family, the alarm may be issued 24 hours a day, regardless of the time, which increases the burden on the family. On the contrary, if the frequency of alarm generation is reduced in order to reduce false positives, the abnormality may be overlooked and the worst result may be obtained. It is extremely important to improve the accuracy of abnormality detection by improving the accuracy of the monitoring monitoring system. Therefore, in the present invention, a thermal imaging device capable of directly detecting the human body is adopted. As a result, the recognition accuracy has been improved, but it is still an important issue to suppress false detections and oversights.

In the future, there is a possibility that AI (artificial intelligence) technology will realize higher accuracy, but at present, it is more accurate for humans to check images, and oversight can be suppressed. Therefore, there is a demand for a service that determines the necessity of dealing with an abnormality when the system detects an abnormality. In this service, when an abnormality is detected in a monitoring system installed in a room such as a nursing facility or in an elderly person living alone, the monitoring service staff does not directly send an alarm to the nursing staff or family members. The data will be sent to. It is a service in which the person in charge of the watching service judges the image and then if the person in charge of the watching service determines that it is necessary, the person in charge of the care service or the family is contacted.

A conceptual diagram is shown in FIG. The watching service staff 23-1 can watch over multiple facilities and houses 24 hours a day. In this case as well, the watching service staff 23-1 does not always watch the image, but when an abnormality occurs, the image before and after the abnormality occurs and the current image are checked, the urgency is judged, and necessary measures are taken. The family member and the guardian 23 do not have to respond to all the alarms generated by the monitoring monitoring system, and only need to be contacted and responded when the monitoring service person 23-1 deems it necessary. As a result, the burden on the family and the guardian 23 can be reduced. For the guardian 23, dealing with midnight and holidays has a large psychological and physical burden, and it is considered that reducing the burden by such a service is particularly effective.

In the case of an elderly person living alone, if the family lives in a remote area, it is often not possible to take immediate action even if an emergency call is received. A rush service has already been provided in case of such a situation. When there is a concern that something is wrong, the rush service person 23-2 rushes to the house where he lives alone and checks the situation on behalf of his family. This rush service and the watching service can be linked. If an abnormality is detected by the monitoring monitoring system and it is determined that the monitoring service requires rescue or confirmation of the target person, the monitoring service staff 23-1 contacts the rushing staff 23-2 and the rush service staff 23- 2 is a service that rushes to the target person's residence.

Since the monitoring service only needs to check the image when an abnormality is detected by the monitoring monitoring system, it is possible to take charge of many facilities at the same time, and it is possible to respond efficiently and promptly. With the aging of society, highly reliable, efficient, and low-cost monitoring and monitoring systems that utilize IoT (Internet of Things) devices are becoming important. Systems that can efficiently support the safe and secure lives of the elderly by utilizing IoT are expected to become more and more important in the future.

Up to this point, we have mainly described the watching and monitoring system installed on the bed in the living room, but the installation location is not limited to this. In long-term care facilities and hospitals, it is effective to install a watching monitoring system on the bed because the person to be watched spends most of the time in a private room or a large room in one room with a bed. On the other hand, elderly people living alone spend more time outside the bedroom. In such a case, it is necessary to watch over the place other than the bedroom. Depending on the location, in the living room, a watching monitoring system using a heat-sensitive imager 10, and in a toilet, corridor, entrance, etc., by combining with a watching monitoring system using a simple pyroelectric sensor, etc., it is possible to watch over the entire house. Reliable monitoring is possible. In the monitoring monitoring system using the thermal image pickup device 10 used in the living room or the like, it is necessary to change the image recognition software. Here, too, the basic configuration is the same: grasping the three-dimensional position of the subject in the room, recognizing the standing, walking, sitting, and lying positions, and monitoring the occurrence of accidents such as falls and the amount of exercise. ..

Although we have shown examples of applications for watching over elderly people and inpatients, the applications are not limited to this. It can also be applied to watching over infants, watching over detainees, and watching over pets. In addition to being used for watching, it can also be used as a surveillance camera to detect humans in streets, stores, and corridors of apartment buildings where ordinary surveillance cameras are used.

Here, application examples other than watching will be explained. Higher-performance control is possible by using the heat-sensitive image pickup device 10 of the present invention for applications such as an automatic lighting switch, an automatic door switch, and an intrusion alarm when an inexpensive pyroelectric sensor has been used in the past. become. For example, in the case of a switch that turns on the lamp when it detects entering the toilet, the pyroelectric sensor detects the movement of the person, so if the person continues to be stationary, the lamp will turn off even while in the room. There was a problem.

If the heat-sensitive image pickup device 10 of the present invention is used, the existence can be detected even when a person is stationary, so that such a problem does not occur. Further, the number of people in the toilet can be measured, and the usage status of all the private rooms in the toilet can be grasped by one heat-sensitive imaging device 10. By sending this information by wireless communication, it becomes possible to grasp the usage status of the toilet outside the toilet. This makes it possible to display the vacancy status of the toilet outside the toilet, and to check the vacancy status on the mobile terminal and select the toilet.

In recent years, services such as recognizing vacant seats in stores such as restaurants with cameras and displaying congestion status and vacant seat information outside the store or on the Internet have become widespread. By using the heat-sensitive imaging device 10 of the present invention when acquiring such vacant seat information in a store, a person can be detected from the body temperature, so that the vacant seat can be grasped more accurately. Furthermore, it is possible to grasp the number of visitors and analyze the customer's staying time and the movement route in the store. It can also be installed in shopping malls and the like to analyze changes in the number of customers and customer flow lines.

For use in stores, it is possible to improve recognition accuracy by inputting seat arrangement information as advance information in advance. By using the heat-sensitive image pickup device 10 of the present invention, there is an advantage that there is no problem with privacy such as image leakage as compared with the method using a video camera.

Further, by utilizing the function of being able to recognize human behavior by image recognition using the thermal image pickup device 10 of the present invention, it can be used as a smart remote controller for home appliances. There are many recent home appliances that can be connected to Wifi or Bluetooth, and the behavior of the target person is grasped using the monitoring monitoring system of the present invention, and the home appliances are appropriate for the user's life by wireless communication. Can be controlled to.

For example, when the subject falls asleep while watching TV on the bed, it is possible to detect falling asleep, turn off the TV, turn off the lights, and so on.

In addition, various uses can be considered, such as controlling the air conditioner according to the state of sleep to adjust the room temperature, and turning off the power of lighting, TV, air conditioner, etc. when the subject goes out of the room.

Although an example has been shown in which the heat-sensitive image pickup device 10 is placed on the ceiling and used for watching and monitoring the room, the installation method of the heat-sensitive image pickup device 10 is not limited to this. It is also possible to invert the positional relationship upside down and install it on the floor or a wall surface near the floor to watch over the room including the ceiling. In monitoring monitoring in an open space without a wall, it is effective to arrange the thermal imaging device 10 on the floor and use it.

(Example 7)
Thermal imaging device for in-vehicle monitoring

Up to this point, examples of an indoor monitoring monitoring system for the elderly and infants have been described, but the application of the monitoring monitoring system of the present invention is not limited to this, and is applied to various other applications. Is possible. Here, as a seventh embodiment, application to a watching monitoring system that is mounted on an automobile to detect a pedestrian will be described.

The required visual field area differs between the in-vehicle monitoring system and the indoor monitoring system. An in-vehicle system for detecting a pedestrian requires a wide field of view in the horizontal direction. On the other hand, a wide field of view is not required in the vertical direction. In order to prevent entanglement when turning left or right, a field of view of 90 degrees (viewing angle 180 degrees) or more is required on each side with respect to the traveling direction of the vehicle. On the other hand, in the vertical direction, a viewing angle of about 30-40 is sufficient. Therefore, it is required to have different viewing angles in the horizontal direction and the vertical direction.

As already mentioned, the conventional far-infrared camera has a narrow viewing angle of 40-50 °, and its performance is insufficient as a pedestrian detection system. Since the catadioptric system using the convex mirror 2 of the indoor watching monitoring system described so far is designed to widen the viewing angle in both the horizontal direction and the vertical direction, the pixels in the vertical direction cannot be effectively used.

Therefore, we have developed an optical system that can widen the viewing angle only in the horizontal direction for in-vehicle use. As already mentioned, the field of view can be expanded by using a convex mirror. By changing the mirror shape in the horizontal direction and the vertical direction, it is possible to control the viewing angle in the vertical direction and the horizontal direction.

Here, the optical system of the thermal image pickup device 10 in which the viewing angle in the horizontal direction is widened by using a cylindrical mirror will be described with reference to FIG. In the optical system shown in FIG. 30, a cylindrical mirror 2-2 is arranged as a convex mirror at an angle of 30 ° with respect to the vertical direction. When the optical axis 3-4 of the cylindrical mirror 2-2 is defined in the outer peripheral direction of the circle, the angle α formed by the optical axis 3-4 of the cylindrical mirror 2-2 and the optical axis 3-3 of the far infrared camera 3 is defined. The far-infrared camera 3 was arranged off-axis so that α = 30 °. Since the curvature of the cylindrical mirror 2-2 in the tubular direction (vertical direction) is zero, the viewing angle in the vertical direction is the vertical viewing angle of the far-infrared camera 3 as shown by the typical ray 6. It will be reflected as it is. Here, a stainless steel cylindrical mirror was used as the cylindrical mirror 2-2. As the far-infrared camera 3, a bolometer-type camera with a viewing angle of 56x42 ° and 160x120 pixels was adopted.

FIG. 31 shows a view of the cylindrical mirror 2-2 as viewed from the upper part of the cylinder. FIG. 31 shows light rays for each viewing angle of 5 ° of the far-infrared camera 3 and light rays reflected by the circular mirror. As can be seen from the light rays reflected by the circular mirror of FIG. 31, it can be seen that the viewing angle is expanded and a wide viewing angle of 180 ° or more is obtained.

It can be seen that by using the cylindrical mirror 2-2, the horizontal field of view can be expanded while maintaining the vertical field of view. The thermal imaging device 10 is such that the far-infrared camera 3 is arranged off-axis with respect to the cylindrical mirror 2-2 so that the far-infrared camera 3 is not reflected in the center of the image.

In FIG. 32, the imaging range of the heat-sensitive image pickup device of the present invention when the heat-sensitive image pickup device 10 for vehicle is attached to the front portion of the roof of the automobile 27 is compared with the case where a normal far-infrared camera is used. Indicated. The area shown by the broken line represents the field of view of a normal far-infrared camera, and the area shown by a solid line represents the field of view when the far-infrared imaging apparatus 10 of FIG. 30 is used. A far wider viewing angle of 220 ° is obtained as compared with the viewing angle of 55 ° of a normal far-infrared camera.

FIG. 33 shows the field of view shown in FIG. 32 with respect to the road. The broken line represents the field of view of a normal far-infrared camera, and the solid line represents the field of view when the thermal imaging apparatus 10 of the present invention is used. With a viewing angle of 55 ° of a normal far-infrared camera, it is difficult to detect pedestrians 28 and bicycles 29 near the automobile 27, and it is not possible to detect pedestrians getting caught when turning left or right or jumping out of an intersection with poor visibility. You can see that. Since these accidents account for most of the accidents resulting in injury or death in urban areas, it is particularly important to detect pedestrians 28 and bicycles 29 in the vicinity of the automobile 27.

It can be seen that the in-vehicle thermal image pickup device 10 of the present invention has a wide viewing angle of 180 ° or more (220 ° in the example of FIG. 33) and can detect pedestrians, motorcycles, and bicycles in the vicinity of the automobile.

A far-infrared image of a person on an actual road is shown in FIG. 34. The vehicle is in the left lane of the two lanes, and the figure shows the center line, the ends of the left and right lanes, and the distance from the vehicle to the pedestrian. Since the car is in the left lane, the left lane is shot in the center of the screen. The distance between the pedestrians on the left and right sidewalks and the car is 50, 20, 10, 5, 2, 0 m. 0 m indicates that there is a pedestrian right next to the thermal image pickup device 10 installed in the vehicle.

Since the cylindrical mirror 2-2 is used, the distortion of the image is large, but it can be seen that the image is taken from a pedestrian at a distance of 20 m from the vehicle to a pedestrian at 0 m (right next to the vehicle). Detection of a pedestrian at a distance of 50 m is difficult from the image of FIG. 34. This is because the resolution of the far-infrared camera used is as low as 160x120 pixels.

If a high-resolution far-infrared camera is used, it is possible to detect pedestrians at a distance of 100 m or more. However, since a high-resolution far-infrared camera is expensive, it is necessary to select a camera according to the purpose. Here, since the detection of pedestrians in the vicinity of the automobile is prioritized over the detection of pedestrians in the distance, a low-priced low-pixel far-infrared camera was selected.

From the image of FIG. 34, a pedestrian is detected by image recognition, and the driver is alerted as necessary. As already mentioned, the far-infrared camera can detect a person directly from the body temperature, so that the person can be easily detected. Similar to image recognition with an elderly person watching and monitoring device, it is possible to identify a person with higher accuracy by the temperature, shape, and size on the image.

When a person is detected around the vehicle, an alarm / warning is given to the driver according to the degree of danger. Warnings to the driver are given by warning sounds, warning lights, and displays installed in the vehicle. For example, the volume and pitch of the alarm sound can be changed according to the degree of danger, or the blinking speed of the alarm light can be changed to give a warning. When using a display, a temperature image can be displayed and a person mark can be displayed on the display to notify the warning.

Up to this point, we have described the detection of pedestrians, but since pets and wild animals on the street also have body temperature, they can be detected from temperature images. It is also possible to distinguish between humans and animals by image recognition and issue different alarms.

In the above embodiment, a cylindrical mirror 2-2 having a circular cross-sectional shape in the horizontal direction and a straight cross-sectional shape in the vertical direction is used as the reflection mirror, but the present invention is not limited to this. For example, the cross-sectional shape in the horizontal direction may be an ellipse, a parabola, or a hyperbola. Further, the cross-sectional shape in the vertical direction can be reduced in the viewing angle in the vertical direction and the magnification can be increased by using a concave mirror. By using the concave mirror, the detection sensitivity of a distant pedestrian can be increased by increasing the magnification at the center of the screen. The concave shape of the lens at this time can also be selected from a circular shape, an ellipse, a parabola, a hyperbolic shape, and the like.

FIG. 35 shows an example in which the saddle-shaped mirror 2-3 is adopted. Here, as the saddle-shaped mirror 2-3, a quadric curved surface called a hyperbolic paraboloid was adopted. A hyperbolic paraboloid is a curve that satisfies the relationship z = (x / a) ^2- (y / b) ² . The cross section in the horizontal direction is a convex mirror, and the viewing angle is enlarged as shown in FIG. On the other hand, the vertical direction is a concave mirror, and the viewing angle is reduced.

The narrow viewing angle means that the image can be captured at high magnification, which means that the detection performance of pedestrians in the distance is improved. By designing a more complicated three-dimensional shape, the magnification can be controlled by the position on the screen. For example, by increasing the curvature of the concave mirror in the vertical direction at the center of the screen and decreasing it toward the outer circumference, the magnification at the center of the screen can be increased.

Since the thermal imaging device 10 is installed in the direction of travel of the automobile, the road in the direction of travel is photographed in the center of the screen. As shown in FIG. 34, a distant pedestrian is always shown in the center of the screen. By increasing the curvature of the concave mirror in the vertical direction at the center of the screen, it is possible to image a distant pedestrian with high sensitivity.

As already mentioned, the cross-sectional shape of the convex mirror in the horizontal direction can be increased in magnification toward the periphery of the screen with respect to the center of the screen by decreasing the curvature from a circle to a parabola, a hyperbola, and the outer periphery. .. In the case of in-vehicle use, the pedestrian in the vicinity of the screen is close to the vehicle, so that the pedestrian is photographed sufficiently large as shown in FIG. 34. In the case of in-vehicle use, it is easier to detect a pedestrian in the distance if the cross-sectional shape of the convex mirror in the horizontal direction is a shape in which the curvature increases toward the outer peripheral portion.

In this embodiment, in order to prioritize the detection of pedestrians in the vicinity of the vehicle, a cylindrical mirror 2-2 having a circular cross-sectional shape in the horizontal direction of the convex mirror was adopted.

A desired viewing angle can be designed by controlling the surface shape of the mirror. Although it is costly to manufacture a mirror having a complicated surface shape, it is possible to manufacture a mirror having an arbitrary shape at a relatively low cost by depositing a metal film on a plastic material which has been injection-molded. Further, since the far-infrared light wavelength is 10 times or more the visible light wavelength, the requirement for mirror processing accuracy can be relaxed by an order of magnitude or more. Another advantage of using a far-infrared catadioptric system is that the manufacturing cost of the mirror can be reduced.

Adopting a high-pixel far-infrared camera to improve the detection performance of pedestrians in the distance will lead to a significant cost increase. On the other hand, changing the mirror shape can be realized at a relatively low cost. The shape of the reflection mirror can be designed according to the performance required for the in-vehicle monitoring device. Further, although the heat-sensitive image pickup device 10 is installed toward the front in the traveling direction here, it is also possible to install the heat-sensitive imaging device 10 toward the opposite side in the traveling direction to detect a pedestrian behind. ..

(Example 8)
Thermal imaging device for driver / passenger monitoring

In the seventh embodiment, an embodiment attached to the front outside of the vehicle to detect a pedestrian on the road was shown. As an eighth embodiment, an embodiment of a thermal imaging device that can be attached to the inside of a vehicle to monitor a driver or a passenger will be described.

In recent years, automobiles equipped with a driver monitoring function have been put on the market. It is a function that takes a picture of the driver using a camera and performs personal authentication of the driver from the image, an alarm when driving aside, an alarm when drowsiness, and an alarm when falling asleep. By personal authentication with a camera, it is possible to automatically switch the seat position, door mirror position adjustment, air conditioner setting, etc. It is common to detect aside driving by identifying the direction of the face, and to take a nap by measuring the degree of eye opening (eye opening degree).

In addition, the development of a system with a function that can monitor not only the driver but also the passengers at the same time is underway. It is a system that can monitor not only the driver but also the passengers with one camera using a wide-angle camera. The passenger's face recognition function makes it possible to identify the passenger, select a favorite music, and operate an in-vehicle device that recognizes hand gestures such as the movement and shape of the passenger's hand.

The problem of driver / passenger monitoring using a conventional camera is the change in the brightness inside the vehicle. The lighting environment inside the vehicle changes drastically from a dark condition at night to a bright condition in direct sunlight. In particular, when a part of the inside of the vehicle is exposed to direct sunlight, a large difference in brightness occurs in the image, which makes it difficult for the camera to recognize it.

In order to avoid this problem, it is common to use near-infrared lighting and near-infrared camera in the in-vehicle monitoring system. However, since the sunlight also contains near-infrared light, there is a problem that the difference between light and dark becomes too large and recognition becomes difficult under the condition of being exposed to direct sunlight.

This problem can be solved by using a far infrared camera. However, as I have repeatedly stated, conventional far-infrared cameras have a narrow viewing angle, and it was not possible to capture the driver and all passengers with a single camera. By using the heat-sensitive image pickup device of the present invention, a wide field of view has been realized and all passengers can be monitored.

FIG. 36 shows an installation example of the thermal image pickup device 10. The thermal image pickup device 10 is installed in the vicinity of the windshield 25 on the rearview mirror 26 of the automobile toward the room. Here, too, the cylindrical mirror 2-2 is used to widen the viewing angle in the horizontal direction. The heat-sensitive image pickup device 10 is attached to the roof 24 portion of the automobile, and the cylindrical mirror 2-2 is vertically attached so that the field of view looks down.

FIG. 37 shows a temperature image of the inside of the vehicle taken by the thermal image pickup device 10 of FIG. 36. The image is reduced in half in the vertical direction for easier viewing. It can be seen that the driver 30 and the three passengers 31 (the passenger in the passenger seat and the two passengers in the back seat) are photographed. Since the person is detected from the body temperature, only the passenger is photographed, and only the person is photographed.

From the image, it can be seen that the driver 30 is wearing glasses. Since the temperature of the glasses is lower than the body temperature, they appear black. It can also be recognized from the image that the driver 30 is raising the index finger of his left hand. From the direction of the driver 30's face, it is possible to prevent the driver from looking away and to operate the audio device by recognizing the shape and movement of the passenger's hand. Even from low-resolution temperature images, personal authentication was possible by utilizing face recognition technology based on machine learning.

By using the heat-sensitive image pickup device 10 of the present invention for in-vehicle monitoring, it has become possible to monitor not only the driver 30 but also the passenger 31. As a result, it is possible to reduce accidents by monitoring the degree of attention and drowsiness of the driver 30. Furthermore, the position and movement of not only the driver 30 but also the passenger are monitored, music suitable for the passenger is selected using the passenger's face recognition function, and the movement and shape of the passenger's hand are recognized by hand gesture recognition. It becomes possible to operate the in-vehicle device. Further, in the thermal imaging device 10, it is possible to control the optimum temperature and air volume of the air conditioner for each passenger from the body surface temperature of the passenger. By enabling fine-tuned control for each passenger, comfort during boarding can be improved. In addition, when a child is placed in the back seat, the behavior and state of the child can be grasped from the driver's seat.

Further, there is another usage for driver monitoring using the thermal imaging device 10 of the present invention. It is an evaluation of driving stress. It is known that driving stress can be evaluated by measuring the temperature change of the nose, and research is underway (for example, Yamakoshi et al., "A trial of driving stress evaluation using differential facial skin radiation temperature". -Basic study based on monotonous driving stress load ", Biomedical Engineering, 48 (2), pp.163, 2010). It is known that driving performance deteriorates when driving stress is too high or too low, and it is important to understand the stress state during driving in order to realize safe driving.

The driver's stress was measured using the thermal image pickup device 10 shown in FIG. 36. Stress causes contraction of peripheral blood vessels, which lowers the skin temperature in the peripheral area. Here, the temperature difference between the forehead surface temperature, which has a strong correlation with the trunk temperature, and the nose temperature, which represents the skin temperature of the peripheral part, was measured to evaluate the driving stress. The driver's forehead and nose areas were recognized by image recognition of the temperature image, and the temperature difference was calculated from the temperature of each part. The facial temperature was evaluated by the thermal image pickup device 10 while running for 60 minutes in the drive simulator. It was confirmed that the nasal temperature gradually decreased by performing monotonous driving. Using the temperature difference between the forehead temperature and the nose temperature as an index, it has become possible to evaluate stress and warn the driver.

(Example 9)
Thermal imaging device for autonomous vehicles

In recent years, the development of self-driving cars has been promoted. Not only automobiles, but also various autonomous vehicles that are not operated by humans are being developed and partially put into practical use. For example, autonomous driving heavy construction machines, transport robots, delivery robots, catering robots, and the like have been put into practical use. In addition, large commercial cleaning robots are also commercially available. Unlike factories, autonomous driving vehicles and autonomous driving robots that operate in an environment where people and pets coexist place importance on safety, especially prevention of contact accidents with people. It is required to automatically recognize people and pets in real time while driving and to perform safe driving.

The heat-sensitive image pickup device 10 of the present invention can be used for such an application. Since people and pets can be selectively detected from body temperature even in an environment where various objects exist, it is possible to drive safely at a distance from people and pets. Unlike obstacles that do not move, people and pets can move suddenly and have a large impact in the event of an accident, so you should drive at a greater distance from people and pets than normal obstacles, or speed up. It is especially important to drop the vehicle to ensure safety. By using the heat-sensitive image pickup device 10 of the present invention, people and pets can be reliably recognized in a wide range in the traveling direction, so that safe autonomous driving can be realized.

FIG. 38 shows an application example to an autonomous driving vehicle for delivery. The thermal imaging device 10 is mounted on the roof in front of the vehicle. Other sensors such as a normal camera and a proximity sensor can be arranged in the vicinity of the thermal image pickup device 10.

The thermal imaging device 10 recognizes people and pets in the direction of travel in real time, and the arithmetic unit connected to the device makes it possible to drive safely while changing the driving route.

The monitoring monitoring system using the thermal imaging device, the thermal imaging device, and the monitoring monitoring method using the thermal imaging device of the present invention are used for monitoring and healthcare of elderly people, pedestrian detection of vehicles, and passengers. Useful for monitoring applications.

1 frame
2 Convex mirror
2-1 Spherical mirror
2-2 Cylindrical mirror
2-3 Saddle-shaped mirror
3 Far infrared camera
3-1 lens,
3-2 Far infrared array sensor
3-3 Exit pupil center point
4 Wiring
5 single board computer
5-1 Accelerator
6 Typical rays
6-1 Rays in the normal direction of the convex mirror
6-2 Rays in the tangential direction of the convex mirror
7 Far infrared window
10 Thermal imaging device
11 Persons to be watched over
12 beds
13 door
14 shelves
15 windows
16 floors
17 desk
18 chairs
19 temperature image
20 host computer
21 Mobile terminal
22 Mobile terminal display contents
23 Parents
23-1 Watching service staff
23-2 Rush service personnel
24 roof
25 windshield
26 Room mirror
27 car
28 Pedestrian

29 bike
30 driver
31 Passenger
32 camcorder
33 Image processing unit
34 Judgment unit
35 Data storage
36 Output section

Claims

It is a heat-sensitive imaging device that senses people from body temperature.
A heat-sensitive image pickup device, characterized in that the image pickup device has a catadioptric system including a convex mirror arranged off-axis.
The heat-sensitive imaging device according to claim 1.
A heat-sensitive image pickup apparatus, characterized in that the convex mirror is a convex mirror having a shape in which the curvature decreases from the axis of rotational symmetry toward the outer circumference, including a parabolic mirror or a hyperboloid mirror.
The heat-sensitive imaging device according to claim 1.
A heat-sensitive image pickup device, wherein the convex mirror has different shapes in the horizontal direction and the vertical direction, and at least one of the horizontal direction and the vertical direction has a convex mirror shape.
The heat-sensitive imaging device according to claim 1.
A heat-sensitive imaging device characterized in that the convex mirror has a cylindrical shape.
The heat-sensitive imaging device according to any one of claims 1 to 4.
Assuming that the viewing angle of the far-infrared camera included in the heat-sensitive image pickup device is θ, the angle α formed by the optical axis of the far-infrared camera and the optical axis of the convex mirror is determined.
A heat-sensitive imaging device characterized in that it is within the range of θ / 2 <α <(π-θ) / 2.
The heat-sensitive imaging device according to any one of claims 1 to 5.
The heat-sensitive image pickup device is characterized in that the heat-sensitive image pickup device is installed near the ceiling of a wall surface closest to the longitudinal direction of bedding such as a bed or a futon in a room.
The heat-sensitive imaging device according to any one of claims 1 to 5.
A heat-sensitive image pickup device, characterized in that the heat-sensitive image pickup device is installed toward the front or the rear in the traveling direction of an automobile or an autonomous traveling vehicle.
The heat-sensitive imaging device according to any one of claims 1 to 5.
A heat-sensitive image pickup device, characterized in that the heat-sensitive image pickup device is installed toward the inside of an automobile.
The heat-sensitive imaging device according to any one of claims 1 to 8.
An image processing unit that analyzes a two-dimensional infrared radiant energy image captured by the heat-sensitive image pickup device and detects the position of the subject on the image and the infrared radiant energy contained in the image.
A determination unit that determines the situation of the subject based on the position on the two-dimensional image and the infrared radiant energy detected by the image processing unit.
A data storage unit that records the status of the target person,
A monitoring monitoring system including an output unit that outputs information indicating the situation of the target person obtained by the discriminating unit.
The monitoring system according to claim 9.
The monitoring unit is characterized in that it determines the current situation of the target person by referring to at least one of the indoor arrangement information or the physical information of the target person input in advance. ..
A monitoring monitoring method using the monitoring monitoring system according to any one of claims 9 to 10.
The image processing unit obtains the temperature or position of the subject's head from the image of the subject included in the temperature distribution image of the omnidirectional heat-sensitive image pickup device, and obtains at least one time variation of the head temperature or the head position. A monitoring monitoring method characterized in that at least one of the subject's respiratory rate, heart rate, and sleep depth is estimated by analysis.