JP6944222B2 - Watching monitoring system - Google Patents

Description

The present invention relates to a thermal imaging device, a monitoring monitoring system using a thermal imaging device, and a monitoring monitoring method using a thermal imaging device.

With the progress of the aging society, the number of elderly people who need long-term care and those who live alone is increasing. Appropriate support is required to protect the daily safety and health of such elderly people. When an accident such as a dementia patient, a resident of a nursing care facility, a hospitalized patient, or an elderly person living alone occurs in a private room due to poor physical condition or a fall or fall, the family or caregiver does not notice and the response is delayed. In such cases, the symptoms become severe and, in the worst case, fatal events occur.

Various private room monitoring systems have been proposed in order to detect illnesses and accidents in such private rooms at an early stage. The elderly watching system includes a method of detecting the state of the person to be watched by using a sensor and a method of observing the state of the person to be watched by an image of a video camera or the like.

In the method using sensors, a technique has been proposed in which a plurality of sensors such as a charcoal infrared sensor, a pressure sensitive sensor, an optical sensor, and an ultrasonic sensor detect the state of a target person and notify the detected state. (Patent Document 1). On the other hand, as a technique for recognizing the state of a subject using an image of the subject, a technique of relaying the image of the subject to a caregiver has also been proposed (see Patent Document 2).

If the biometric information and behavior of the person to be watched can be grasped by such an elderly person watching and monitoring system, it is expected to be useful not only for ensuring the safety of the elderly but also for health promotion and health care.

Although the number of fatalities in traffic accidents is declining, it still exceeds 3,000 a year. A characteristic of traffic fatalities in Japan is the high rate of accidents while walking or riding a bicycle. In particular, pedestrian and bicycle accidents increase at night. This is because it is harder to find pedestrians and bicycles at night than at noon. In order to solve such a problem, a vehicle equipped with a far-infrared camera (thermography camera) capable of detecting a pedestrian from body temperature even at night has been developed and has already been sold (see Patent Document 3).

Japanese Unexamined Patent Publication No. 2001-327549

Japanese Unexamined Patent Publication No. 2000-105885

Japanese Unexamined Patent Publication No. 60-231193

Among conventional elderly watching systems, single-function sensors such as pressure sensors and optical sensors are used to detect specific movements such as getting up and getting out of bed, and when such movements are performed, Generate an alarm. With such a simple sensor, it is not possible to determine the detailed situation of the person to be watched over, so it is difficult to accurately determine whether the situation really requires assistance. For this reason, there is a problem that many unnecessary alarms are generated and the burden on the caregiver is increased.

In addition, depending on the sensor settings, the behavior to be detected was often overlooked. As a method for monitoring the behavior of the subject in more detail, a behavior monitoring system using a video camera has been put into practical use. However, surveillance using a video camera has a large privacy problem and has not been widely used in the living space of the elderly. This is because there are problems such as a large psychological resistance of the person being watched over, a large physical and psychological burden on the caregiver to confirm the image, and a risk of leakage of images and the like. Further, if the image is simply relayed, it is necessary to constantly monitor the image, which causes a problem that the burden on the caregiver is heavy and there is a risk of overlooking.

In order to solve such a problem, a system has been proposed in which an image recognition program is used to determine and warn the subject's unusual behavior or dangerous state. However, even with such a system, it is difficult to appropriately detect an accident or abnormality of the subject from a complicated image, and there is a problem that an oversight or a false alarm occurs. In addition, after detecting an abnormal state, it is necessary for the caregiver to judge the necessity of caregiving by looking at the actual image, and there is also a problem that a privacy problem occurs here.

Furthermore, when watching over using a video camera, a high-resolution image is required in order to recognize the target person and his / her behavior in the presence of various objects in the room. The use of high resolution images causes privacy issues. In addition, a computer with a large computing power is required for image recognition for analyzing a high-resolution image with a computer and recognizing the position and behavior of the person being watched over. For this reason, in a monitoring monitoring system using a video camera indoors, an image is often sent to a host computer outside the private room where image monitoring and image recognition are performed. A high-priced image transmission device and a host computer are required for relaying such images, and at the same time, there is a risk of image leakage.

There is a need for a monitoring monitoring system that can accurately grasp the current situation of the target person while giving consideration to privacy, accurately detect the situation requiring assistance, and issue an alarm. In addition, it is desirable that the watching monitoring system is small and arranged so as not to interfere with the daily activities and assisting movements of the monitoring target person and not to interfere with the daily life of the target person and the caregiver. Furthermore, it is desirable that the system has a simple configuration and is inexpensive so that the cost of nursing care and watching can be suppressed.

Next, the problems of the existing in-vehicle monitoring monitor system will be described. The problem with in-vehicle surveillance systems using existing far-infrared cameras is that the viewing angle is narrow. The viewing angle of an in-vehicle far-infrared camera currently on the market is 40 degrees or less, and it is a system for detecting pedestrians and animals in the distance. This is because the field of view of the conventional far-infrared camera is narrow and it is not possible to realize a wide field of view.

Accidents between cars and pedestrians / bicycles often occur in the city, especially when turning left or right, often due to entanglement or oversight of pedestrians crossing. In order to prevent these accidents, it is important to be able to detect pedestrians and bicycles in the vicinity of the vehicle. For that purpose, it is important to be able to monitor the vicinity of the vehicle with a wide field of view.

An object of the present invention is to provide a thermal imaging device, a watching monitoring system, and a watching monitoring method capable of detecting a target person in a wide field of view with a simple configuration.

In order to solve the above-mentioned problems, in the present invention, monitoring and monitoring are performed using a far-infrared camera (thermography) capable of photographing in a wide viewing angle. A far-infrared camera is a device that captures a temperature distribution calculated from the energy of far-infrared rays emitted from an object as an image. Since humans have a body temperature, which is higher than the normal ambient temperature, humans can be selectively detected from the body temperature. Therefore, unlike a normal video camera, it is not necessary to acquire a high-resolution image in order to recognize a human from the image, and a human can be selectively detected from a low-resolution temperature image, so that there are few recognition errors. The computing load on the computer can be reduced. Therefore, processing can be performed by a computer having a small computing power, and the system can be reduced in price and size.

Since a person can be detected directly from the body temperature, a monitoring device using a far-infrared camera has been conventionally proposed. The problem of the monitoring monitoring system using such a far-infrared camera is that the imaging range of the far-infrared camera is narrow. Since the viewing angle of a commercially available far-infrared camera is about 40 to 60 °, it is difficult to monitor and monitor a wide range. For example, as an example of an elderly person watching system, there is "Non-patent document: Takagi et al." Watching elderly people using a temperature sensor ", DEIM2016, P6-1, 2016". In this system, a system has been proposed in which a far-infrared camera is installed on the ceiling on the bed or on the wall surface facing the bed to monitor the behavior on the bed (such as detection of a fall from the bed). In such a system, there is a problem that the watching range is limited to the bed.

An example of forward monitoring of an automobile is a night vision system using an Autoliv far-infrared camera mounted on the BMW 7 Series. In this system, the viewing angle is set to 36 degrees during normal driving and 24 degrees during high-speed driving. Therefore, although it was possible to detect pedestrians and animals in the distance, it was not possible to detect pedestrians in the short distance. This is also because the field of view of the conventional far-infrared camera is narrow and it is not possible to realize a wide field of view.

In order to solve the problem that the field of view of a far-infrared camera is narrow and it is possible to monitor and monitor only a limited area, research is being conducted on scanning a far-infrared camera with a motor to take a wide range (non-patent documents). : Shimamura, "Understanding the Position and State of People Using Spatial Sensors", Bulletin of Hosei University Graduate School, 5, 2016-03-24). In such a method, since the device includes moving parts, the device is complicated and there is a concern that a failure may occur, and a process of joining images is required, and it takes time to acquire an image of the entire area, which causes an emergency situation. There was a risk of overlooking.

An example of realizing a monitoring device with a wide viewing angle by combining a far-infrared camera and a curved mirror arranged on the optical axis has also been reported (Non-patent document: Wong Wai Kit et al., "Omnidirectional Thermal".
Imaging Surveillance System Featuring Trespasser and Faint Detection ”,
International Journal of Image Processing, Vol 4 (6), pp.518 (2011).). In this example, a hyperboloid mirror is arranged on the optical axis of the far-infrared camera to realize a wide field of view. This method has a problem that a blind spot is generated in the center of the visual field.

If a far-infrared camera capable of simultaneously capturing the entire room area without blind spots can be realized, these problems can be solved and a practical far-infrared camera can be realized. In the present invention, by using a heat-sensitive image pickup device having a reflection-refraction type optical system off the optical axis, a heat-sensitive image pickup device having an imaging range in a wide area without a blind spot is realized, and a monitoring monitoring system using this device is realized. Was realized.

A wide imaging range can be realized without blind spots by using a thermal imaging device having an optical system that is off-axis and reflects and refracted. Since the heat-sensitive image pickup device can selectively detect a human from a low-resolution image, there is an advantage that image recognition becomes easy and recognition accuracy can be improved.

Thermal imaging device Conventional far-infrared camera with a wide field of view Explanatory drawing showing the imaging range of a catadioptric system Explanatory drawing showing a cross-sectional shape of a convex mirror Figure comparing spherical mirror and parabolic mirror Overall configuration diagram of the elderly watching system installed in the living room of the elderly facility Top view of the indoor installation layout of the thermal imaging device Indoor installation bird's-eye view of thermal imaging device Two-dimensional temperature image of the room Explanatory drawing showing the required viewing angle of the thermal image pickup device Indoor imaging temperature image of thermal imaging device Indoor captured image of visible light camera Explanatory diagram of initial settings for room layout and target audience Relationship between head size and head position Relationship between the distance between the thermal imaging device and the head and the measured temperature Bed movement and sleep depth measurement results Actual measurement data of chest position by head temperature and respiration Results of frequency analysis of chest position by head temperature and respiration Pictogram display example of posture recognized from temperature image and temperature image Posture and situation of the person to be watched, pictogram display example when an alarm occurs Thermal imaging device with a design different from that of FIG. Flow chart of watching monitoring system Conceptual diagram of a monitoring system for the elderly living alone. Conceptual diagram of a monitoring system for nursing care facilities and hospitals Daily life analysis data using the monitoring monitoring system Monthly life analysis data using a watching monitoring system Annual life analysis data using a watching monitoring system Monthly walking count analysis data using a monitoring monitoring system Conceptual diagram of watching service and rush service Thermal imaging device using a cylindrical mirror Explanatory drawing showing the horizontal imaging range of a cylindrical mirror Explanatory drawing which shows the imaging range of an in-vehicle thermal image pickup apparatus The figure which shows the imaging range on the road of the in-vehicle thermal image pickup apparatus Temperature image of pedestrians on the road using an in-vehicle thermal imager Thermal imaging device using a saddle-shaped mirror Application example of thermal imaging device to driver and passenger monitors Temperature image inside the car Application example of thermal imaging device for autonomous vehicles for delivery

About thermal imaging device for watching

The heat-sensitive image pickup apparatus 10 having the entire room as the imaging range will be described with reference to FIG. The heat-sensitive image pickup device 10 is a single board computer 5 connected to a far-infrared camera 3 including an image pickup lens 3-1 and a far-infrared array sensor 3-2, a convex mirror 2, a far-infrared camera 3, and a wiring 4. Is integrated by the frame 1. Here, the characteristic is that the optical system of far-infrared light is an off-axis reflection-refraction type optical device that combines an image pickup lens 3-1 that utilizes refraction and a convex mirror 2 that utilizes reflection. be. Here, the optical axis off-axis optical system means that the optical axis 3-4 of the convex mirror 2 is inclined with respect to the optical axis 3-3 of the far-infrared camera 3. The convex mirror 2 has an axisymmetric three-dimensional shape. It can be seen that by using the reflection of the convex mirror 2, the viewing angle of the far-infrared camera 3 is expanded so that a wider range can be imaged. When the heat-sensitive image pickup device 10 is installed on a wall surface near the ceiling, it can be seen that a wide field of view including the ceiling region and almost the entire wall surface region can be obtained as shown by the typical light beam 6.

Since the convex mirror 2 has a rotationally symmetric shape, it has a wide field of view even in the depth direction of the paper surface of FIG. As a heat-sensitive imaging device that realizes a wide imaging range by utilizing the reflection of the convex mirror 2, the convex mirror 2 as shown in FIG. 2 is arranged on the far infrared camera 3 with the optical axis aligned, and is far away. An attempt to make the infrared camera 3 omnidirectional (360-degree field of view) has been reported (Non-Patent Document: Eiichi Narimatsu, "Development of an omnidirectional thermograph for finding and tracking a person indoors", Proceedings of the 21st Annual Meeting of the Robotics Society of Japan, 2003). In the example of FIG. 2, a hyperboloid mirror is used as the convex mirror 2. In an optical system having such a configuration in which the optical axes of the convex mirror 2 and the far-infrared camera 3 are aligned with each other, there is a problem that a blind spot occurs because the far-infrared camera 3 is reflected in the center of the field of view. There was a problem that it could not be used for watching purposes.

In order to solve this problem, in the present invention, the far-infrared camera 3 is arranged at an angle with respect to the convex mirror 2, and as shown in FIG. 1, a blind spot is provided by using a reflection-refraction type optical system off the optical axis. A wide range of images can be acquired without it. It can be seen that by using the convex mirror 2, the viewing angle is wider than the viewing angle of the far infrared camera 3. Further, the catadioptric system using the convex mirror 2 will be described in detail with reference to FIG. The optical axis 3-3 of the far-infrared camera 3 is tilted with respect to the optical axis 3-4 of the convex mirror 2.

At this time, the effective imaging range of the convex mirror 2 is the light rays 6-1 in the normal direction of the convex mirror 2 passing through the ejection pupil center point 3-5 of the imaging lens 3-1 and the ejection pupil of the imaging lens 3-1. It is within the range of the light beam 6-2 in the tangential direction of the convex mirror 2 passing through the center point 3-5.

Assuming that the angle formed by the optical axis 3-3 of the far-infrared camera 3 and the optical axis 3-4 of the convex mirror 2 is α and the total viewing angle of the far-infrared camera 3 is θ, the direction of the normal direction of the convex mirror 2 is In order for the ray 6-1 to be included in the field of view, θ / 2 <α
It is necessary to satisfy the condition.

On the other hand, in order for the light beam 6-2 in the tangential direction of the convex mirror 2 to point upward from the horizontal direction, θ / 2 <.
It is necessary to satisfy the condition of π / 2 -α. This is to prevent a blind spot from being generated on the wall surface facing the wall surface on which the thermal image pickup device 10 is installed when the tangential light beam 6-2 is directed downward from the horizontal direction.

In order to satisfy both of the above two conditions, the angle α formed by the optical axis 3-4 of the convex mirror 2 and the optical axis 3-3 of the far-infrared camera 3 is θ / 2 <α <(π-θ) /. It is necessary to satisfy the relationship of 2. The viewing angle of the far-infrared camera 3 may differ between the horizontal direction and the vertical direction, in which case θ corresponds to the smaller viewing angle.

Next, the shape of the convex mirror 2 will be described. A wide viewing angle can be obtained by using the convex mirror 2, but the distortion of the image becomes large. By changing the mirror shape, it is possible to control the distortion characteristics of this image. For example, it becomes possible to enlarge the area of interest in an image and shoot it.

It is important to increase the resolution on the bed for watching over the elderly. This is because elderly people spend a lot of time on the bed, and it is important to acquire sleep data in order to understand their health condition. By increasing the magnification of the part where the bed is photographed in the image pickup screen of the thermal image pickup device 10, sleep can be converted into data with high accuracy.

As the cross-sectional shape of the convex mirror 2, as shown in FIG. 4, a flat ellipse, a circle, an ellipse, a parabola, a hyperbola, and the like can be considered. The curves shown in the figure are compared with figures in which the curvature at X = 0 is constant. In order to compare the effects of the mirrors, the reflected rays of the spherical mirror and the parabolic mirror are shown in FIG. The solid line shows the spherical mirror, and the broken line shows the reflected light beam of the parabolic mirror. The rays show the reflection of the rays from the far-infrared camera at every 2.5 ° viewing angle on the mirror.

Comparing the reflected light rays of the spherical mirror and the parabolic mirror, the light rays reflected at the lower part of the convex mirror (around X = 0) are almost the same, but between the reflected rays of the parabolic mirror as it goes to the outer periphery of the mirror. It can be seen that the angle of is smaller (the magnification is higher). It can be seen that the relative magnification in the screen can be controlled by changing the shape of the convex mirror.

Optical systems using mirrors have been extensively studied in ordinary visible light cameras. In recent years, as for refracting lenses for visible light and near-infrared light, optical material technology and lens processing technology have been rapidly advanced. For this reason, the performance, miniaturization, and low price of lenses with various characteristics have made rapid progress. As for fisheye lenses and ultra-wide-angle lenses that realize a wide field of view, high-performance lenses are available at low prices, and reflective optics are rarely used except for special applications.

On the other hand, in far infrared rays, special materials such as germanium, silicon, and chalcogenide are required for the lens material, the lens is expensive, and the market is small, so it is difficult to obtain a special lens such as a wide-angle lens. In the present invention, a heat-sensitive image pickup device having a wide field of view by combining a standard far-infrared camera (viewing angle of about 50 °) that can be obtained at a low price and a small size with a metal convex mirror that is easy to process and can be obtained at a low price. 10 is realized.

(Example 1)
As the first embodiment of the monitoring monitoring system using the heat-sensitive image pickup device 10 having the optical system of the optical axis off-axis reflection / refraction type, the elderly such as nursing homes, elderly housing with services, special elderly nursing homes, and group homes. The elderly watching system installed in the living room of the facility will be explained. The overall configuration of the elderly watching system is shown in FIG. The elderly watching system has two functions: a watching function that monitors the safety of the daily life of the elderly, and a healthcare function that uses the living data of the elderly acquired by the system to improve the health of the elderly. ing.

The thermal imaging device 10 of the present invention is installed in a private room which is a living space for the elderly. When the system detects that an accident such as a fall or a fall or an abnormality such as a sudden change in physical condition has occurred, the occurrence of the abnormality is sent to the mobile terminal 21 of the caregiver or guardian 23, and the caregiver or guardian 23 rushes in. And protect the safety of the elderly. By accumulating daily life data obtained by watching over and analyzing it as big data accumulated on the cloud, we analyze short-term changes in physical condition, long-term health condition, physical strength, and cognitive function of the elderly. However, by giving appropriate advice, long-term health promotion (health care) for the elderly will be realized.

The monitoring monitoring device using the heat-sensitive image pickup device 10 off the optical axis and the reflection / refraction type shown in FIG. 1 will be described in detail below. The far-infrared camera 3 forms an image of far-infrared rays emitted from an object on a far-infrared array sensor 3-2 arranged in a two-dimensional manner by a refracting lens 3-1 for far-infrared light. It is a device that converts the output signal for each pixel into temperature and displays the temperature distribution as an image. The heat-sensitive image pickup device 10 used here includes a far-infrared camera 3 composed of a microbolometer-type 80x60 pixel far-infrared array sensor 3-2, a silicon refraction lens 3-1 and a convex mirror 2. Will be done. As the convex mirror 2, an aluminum-polished mirror having a rotating paraboloid shape was used. The imaging range of the far-infrared camera 3 used is 51x38 ° in terms of viewing angle, and this viewing angle is magnified by the convex mirror 2 for imaging.

The far-infrared camera 3 is connected to the single board computer 5 by wiring 4. The single board computer 5 used here is equipped with a 1 GHz single core CPU chip and a Wifi communication chip, and processes the temperature image acquired by the heat-sensitive camera to recognize the behavior of the target elderly person, and it is necessary. In response, information can be transmitted to an external computer or mobile terminal (tablet, smartphone, etc.) by Wifi communication.

Since the single board computer 5 is small and inexpensive, it can be integrated with the far infrared camera 3 and installed in the room of the person to be watched over. The single board computer 5 can directly receive the temperature image output signal of the far infrared camera 3 via the wiring 4 and perform image recognition. Edge computing is the process of performing image processing inside a device using an IoT device that consists of a sensor device and a computer with a built-in communication function as in this example. Edge computing has the advantages of reducing communication load, real-time data processing, and reducing security risks, and is a technology that has been attracting attention in recent years. In the monitoring monitoring system of the present invention, the low pixel image of the far-infrared camera is used to reduce the load of image processing and enable edge computing on the single board computer 5 having low computing power.

The monitoring monitoring system of the present invention is not limited to edge computing, and the arithmetic unit may be an ordinary computer and may be installed in a place (for example, a caregiver's station) away from the place where the thermal imaging device is installed. It is possible.

In this case, the image processing unit, the discrimination unit, the data storage unit, and the output unit are placed on the computer side, and perform data communication with the far-infrared camera 3 by wireless or wired communication. From the viewpoint of system price reduction and miniaturization, it is efficient to use a single board computer 5 having a communication function, but if a high-performance computer is required for purposes other than watching, edge computing You can use a normal computer without doing this. For example, it is effective to separate the image processing computer from the edge device when performing complicated operations such as automating the input to the long-term care diary based on the monitoring result.

Here, a parabolic aluminum polishing mirror was used as the convex mirror 2. The aluminum polishing mirror was created by pressing an aluminum plate into a parabolic shape and polishing the surface. Here, a metal mirror is used, but the present invention is not limited to this, and a mirror in which a metal film such as silver or aluminum is vapor-deposited on a polymer material (polycarbonate, cycloolefin, etc.) that has been injection-molded into a parabolic shape can also be used. can. By using a mirror in which metal is vapor-deposited on plastic, the manufacturing cost of the mirror can be reduced. Further, here, the convex mirror 2 uses a parabolic mirror, but a spherical mirror, a conical mirror, and a hyperboloid mirror can also be used.

FIG. 7 shows an installation diagram of the thermal image pickup device 10 in a room. There is a person to be watched 11 in the room, and a bed 12, a door 13, a shelf 14, a window 15, a desk 17, and a chair 18 are arranged on the floor 16. The thermal image pickup device 10 is arranged near the ceiling of the wall surface closest to the longitudinal direction of the bed 12. FIG. 8 shows a bird's-eye view of the arrangement in the room. The three-dimensional coordinates of the head are written as (X0, X0, Z0), and the head temperature is written as T0.

FIG. 9 shows a temperature image taken in the room shown in FIG. 7. Since it is an image reflected by a curved mirror, it is an image with a large distortion, but the floor 16 and the four walls surrounding the bed 12 are imaged. The four walls are photographed almost up to the ceiling, so you can understand the wide viewing angle. Since the person to be watched 11 has a higher temperature than the surroundings, it can be clearly identified. Despite the low resolution (80x60 pixels) image, the head, body, and legs are clearly separated, from which the posture of the subject 11 can be determined. The head is indicated by a cross mark in the image. Here, the coordinates on the temperature image of the central part of the head are (x0, y0), and the measured temperature of the head is t0.

It should be noted in the temperature image of FIG. 9 that the existence and posture of the subject 11 can be recognized, but the individual identification and detailed movement of the subject 11 cannot be identified, and the risk of privacy invasion is low. When a normal camera is used, a high-resolution image is required to recognize a human with high accuracy, and the face, facial expression, and action of the subject 11 are imaged, so that the psychological resistance of the watching subject is increased. growing. In addition, in high-resolution images, identification of objects placed in the room and clutter in the room are also observed, which also increases psychological resistance. The greatest merit of using the heat-sensitive image pickup device 10 for watching over is that the heat-sensitive image pickup device 10 can image the heat generated by the human body, so that the human body can be detected with high accuracy with a low-resolution image in consideration of privacy.

FIG. 9 shows the upper part of the door 13, the upper part of the window 15, and the main body of the thermal image pickup device 10 installed near the ceiling, and it can be seen that the images are taken from the floor 16 to the vicinity of the ceiling without blind spots. Since the entire wall surface including the door 13 and the window 15 is photographed, the position of the person to be watched in the room 11 can be grasped without a blind spot, and it is possible to detect going out from the door 13 to the outside without overlooking.

Here, the reason why the heat-sensitive image pickup device 10 is arranged near the ceiling of the wall which is in close contact with the long side side of the bed 12 is to watch the watch target person 11 on the bed 12 with higher accuracy. Priority was given to watching over bed 12 when the subject 11 was watching over nursing care facilities, hospitals, etc., (1) staying on bed 12 for a long time, and (2) falling from bed 12 or getting out of bed. This is because it is considered that there are many accidents such as falls, and (3) grasping the sleep state is effective for grasping the life and health state of the subject 11. The arrangement position of the thermal image pickup device 10 is not limited to the bed 12, but fixing the arrangement is also advantageous for improving the accuracy of image recognition. For example, if the approximate position of the bed 12 is known in advance on the screen, it is easier to recognize the behavior of the subject 11 such as being in bed, getting out of bed, getting up, etc., as compared with the case of performing image recognition without such information. And it will be possible to do it with high accuracy

Further, in this embodiment, in order to measure biological information (sleep depth, respiration, heartbeat, etc.) during sleep, the imaging magnification on the bed 12 can be increased as compared with the spherical mirror as shown in FIG. A parabolic mirror is used. The convex mirror 2 is not limited to a parabolic mirror, and an ellipsoidal mirror and a hyperboloidal mirror can also be used. Further, since the spherical mirror has an advantage that it is easily available and low in cost, it is possible to use a spherical mirror depending on the application.

By using the off-axis reflection / refraction optical system using the parabolic mirror of the present invention, it is possible to observe in detail an object (here, the bed 12) located diagonally below the camera installation position. This is a great advantage over the case of using a refracting wide-angle camera. When the imaging range extends to the upper part of the wall surface with an optical system using a refraction type wide-angle lens, the bed 12 placed diagonally below the camera appears small in the entire screen, so the situation of the subject 11 on the bed 12 can be seen. It becomes difficult to observe in detail.

Here, the viewing angle required for the thermal imaging device 10 for watching will be described. The watching imager is required to have the ability to watch over the subject 11 in the entire area of the room without blind spots. Going out of the room of the monitoring subject 11 may lead to a serious accident (for example, wandering of a dementia patient) depending on the medical condition and condition of the subject 11, and reliable detection without overlooking is required.

FIG. 10 shows the positional relationship between the subject 11 and the thermal imaging device 10 when going out of the room. Here, it is assumed that the height of the subject 11 is 170 cm, the ceiling height is 240 cm, the room size is 8 tatami mats (13 square meters), and the installation height of the thermal image pickup device 10 is 230 cm. At this time, if there is an imaging range up to a height of 100 cm from the floor surface at the position of the door 13, the buttocks of the subject 11 are caught in the vicinity of the door 13 at the edge of the room, and the behavior of the subject 11 disappearing from the door 13. Can be captured. It can be seen that the risk of overlooking is almost eliminated if an image can be taken up to 150 cm from the floor surface. Even if the tall subject 11 is taken into consideration, if the height from the floor surface to 190 cm is imaged, the head is captured and there is no possibility of overlooking. Therefore, the viewing angle of the wall surface of the thermal imaging device 10 needs to be at least 100 cm or more from under the floor, and it is possible to image an area of preferably 150 cm or more from the floor surface, and particularly preferably 190 cm or more from the floor surface. It is required to be. Under the above conditions, the total viewing angle (2β) required for the thermal imaging device 10 is 140 ° when the imaging range is 100 cm or more from the floor surface, 161 ° at 150 cm, and 167 ° at 190 cm.

There is no inexpensive far-infrared camera with such a wide viewing angle. This is the reason why a new catadioptric system for far infrared rays is adopted in the present invention.

FIG. 11 shows an image of the temperature captured by the thermal image pickup device 10, and FIG. 12 shows a comparison of images captured by a normal video camera. In both cases, the watching subject 11 photographed the scene exiting from the door 13. Here, the optical system of the video camera is not a wide-angle refraction lens normally used, but an optical device corresponding to the same off-axis reflection refraction type visible light as the heat-sensitive image pickup device 10 is prototyped and photographed. 11 and 12 are images of the same scene captured so as to have substantially the same field of view.

In both images, it can be seen that the entire area of the room centered on the bed 12 was taken without blind spots. The resolution is 80x60 pixels for the temperature image and 1920x1080 (HD) for the video camera. Although the video camera image shown in FIG. 12 is taken at a higher resolution, it is not easy to visually recognize a person in the image because various objects having different brightness are shown.

On the other hand, in the temperature image taken by the thermal image pickup device 10 shown in FIG. 11, it can be seen that the human body can be easily detected because the heat generated by the human body can be captured. The advantage of watching over can be understood from the comparison of these two images. It is difficult for a normal visible camera to recognize the target person 11 by image processing and grasp its behavior. On the other hand, from the temperature image of FIG. 11 by the thermal image pickup device 10, the target person 11 can be easily monitored from the body temperature.

About behavior recognition technology from temperature images

Hereinafter, an image recognition technique for detecting the watching target person 11 from the temperature image and recognizing the behavior thereof will be described. In image recognition, it is known that it is effective to limit the objects to be imaged in order to improve the recognition accuracy. By limiting the objects appearing in the image to be recognized and by limiting the imaging range, lighting conditions, and imaging conditions, the image recognition accuracy can be improved.

By fixing the imaging device and imposing restrictions on the imaging conditions, image recognition becomes possible with high accuracy. In the present invention, the dedicated heat-sensitive image pickup device 10 whose imaging range is a wide area without blind spots is arranged so that the installation location is limited to the vicinity of the ceiling of the wall near the bed 12, and image processing is performed on the output image. High recognition accuracy was achieved by optimizing.

A wide viewing angle can be obtained by using a convex mirror, but there is a problem that the image has a large distortion. By performing image processing according to the characteristics of this image distortion, the influence of the image distortion can be reduced and the recognition accuracy can be improved. A heat-sensitive image pickup device with a special optical system called a dedicated far-infrared off-axis reflection / refraction optical system with a wide field of view is placed in a limited installation location to perform image processing according to the characteristics of the image to be captured. By doing so, a high-performance monitoring and monitoring system was realized.

The image recognition method will be described below. By restricting the imaging device and its arrangement method as described above (input image restriction), it becomes easy to accurately grasp the behavior of the target person. Furthermore, by utilizing the information about the imaging target to be grasped in advance (prior information), it becomes easier to grasp the behavior. At the time of startup after the installation of the thermal imaging device, information on the layout of the room and information on the person to be watched are acquired in advance, and image recognition is performed using that information to accurately monitor the behavior of the person to be watched. Made it possible to grasp.

An example of inputting prior information in the initial setting after installation of the thermal image pickup device will be described with reference to FIG. The setting is performed by an external computer using the communication function of the single board computer 5 of the thermal image pickup device 10. First, enter the room area. On the selection screen, enter the room size from 4.5 tatami mats (7.3 square meters), 6 tatami mats (9.7 square meters), and 8 tatami mats (13 square meters). If there is not enough space for the selection screen, keyboard input is performed in square meters. The size of the room may be an approximate area.

Next, the room shape based on the bed is input. If the room is rectangular, enter a horizontal rectangle if the longitudinal direction of the bed matches the longitudinal direction of the room, and enter a vertical rectangle if the directions are orthogonal. Next, the bed position is input. Enter the position of the bed on the wall surface close to the bed using A to C. Then, the door position is selected from 1 to 9 and input. Further, the installation height of the thermal image pickup device 10 and the height of the bed are input. Finally, input information about the person to be watched over. Here, the gender, age, height, and weight of the subject are input.

With the above initial settings, the positional relationship between the thermal image pickup device 10 in the room of FIG. 7, the bed 12 and the door 13 was monitored and input to the monitoring system. By grasping the size and shape of the room, it is possible to recognize the position and posture of the watching target person 11 with high accuracy. Further, by grasping the position of the door 13, it is possible to improve the recognition accuracy of the entrance / exit room.

After the initial setting is completed, the position detection algorithm of the target person 11 is started. The thermal image pickup device 10 is set to output 5 temperature images per second (5 FPS). The image file is set to output 16-bit 80x60 pixel data in PGM format. The image recognition algorithm will be described with reference to FIG. The output temperature image 19 is shown in the upper left of FIG.

First, the head is detected from the temperature image 19. Usually, the head has the highest temperature in the temperature image 19, and there is little problem even if the highest temperature portion in the temperature image 19 is recognized as the head position. However, for example, when a pot or hot food or beverage is placed indoors, these hot objects may be mistakenly recognized as the head. To avoid this, the head is recognized from the size and temperature of the detected high temperature region. The temperature of the head is measured in the range of 28-36 ° C., depending on the distance between the head and the thermal imaging device 10 and the position of the head in the field of view to be imaged.

As the distance between the subject 11 and the thermal imaging device 10 gets closer, the size of the head to be imaged increases and the temperature also rises. The size and temperature of the head also change depending on the position of the temperature image 19 in the field of view. There is a relationship between the position in the visual field, the size of the head, and the temperature as described later, and by recognizing the head using this relationship, it is possible to detect the head with high accuracy. The surface temperature of the head is usually measured at 28-36 ° C., and the larger the size of the head (that is, the closer the head is to the thermal imaging device 10), the higher the temperature. By recognizing the head using this relationship, it has become possible to recognize the head even when other high-temperature objects are included in the screen.

The human head is detected from the temperature image 19, and the coordinate position (X0, Y0, Z0) of the head in real space is estimated using the coordinate position (x0, y0) on the image and the measured temperature t0. (Fig. 14). The spatial coordinate position of the head is estimated using the size of the head and the temperature of the head. First, the method of obtaining the head coordinates from the size of the head will be described. A threshold value is set from the temperature image 19 to determine the region of the head, and the semi-major axis radius (a) and the semi-minor axis radius (b) are obtained by elliptical approximation. From the head area S (= πab), use the average radius r = sqrt (S / π) of the head. The major axis dimension of the head of an adult male is 24 cm, the minor axis dimension is 16 cm, and the average height is 170 cm. Since the heat-sensitive image pickup device 10 used in the present invention has a large image distortion, a spherical plastic container having a radius of 10 cm containing hot water (35 ° C.) is placed on the floor as a simulated head for data acquisition, and a screen is displayed. Data on the top position and the size of the captured image were acquired.

A method of obtaining the three-dimensional spatial position of the head from the dimensions of the head on the screen will be described with reference to FIG. In FIG. 14, the position is shown on the xz plane for simplicity, but the same calculation can be performed on the xyz space. From the captured temperature image 19, the center position x0 of the head and the area S0 can be obtained. From the position x1 and area S1 of the sphere with a radius of 10 cm measured on the floor surface and the area S0 actually imaged, the actual head position x0 can be obtained as x0 = sqrt (S1 / S0). Further, the height z0 of the head is obtained as z0 = (1-sqrt (S1 / S0)) * H, where H is the installation height of the thermal imaging device 10.

Next, a method of calculating the coordinate position from the head temperature will be described. In the thermal imaging device 10, the measured temperature is affected by the distance. This is mainly because the area occupied by the target object in the pixels becomes smaller as the distance increases, and it becomes affected by the ambient temperature of the measurement object. This temperature change also changes depending on the position of the temperature image 19.

In FIG. 15, the measured temperature of a spherical plastic container having a radius of 10 cm containing hot water (35 ° C.) at the positions of the center of view (A) and the edge of the field of view (B) is set to the distance between the heat-sensitive image pickup device 10 and the plastic container. It was plotted against it. The distance between the thermal imaging device 10 and the head can be estimated from the pixel position and the measured temperature of the head. If the distance is known, the head position (X0, Y0, Z0) in the space can be obtained from the relationship shown in FIG. As described above, the position of the head in the space can be obtained from both the head size data and the head temperature data. By combining these two measured values, the head position can be measured with high accuracy.

If the position of the head of the subject 11 is known, the posture and behavior of the subject 11 can be estimated. For example, assuming that the height of the bed 12 is 40 cm, it can be estimated that the subject 11 is in the recumbent position if the head is on the bed 12 and the height of the head is within the range of 40-70 cm. If the head is on the bed 12 and the height is within the range of 80-140 cm, it can be estimated that the subject 11 is in the long sitting position. If the head is near the edge of the bed and the height is within the range of 80-140 cm, it can be estimated that the subject is in the sitting position. Since the sitting position is regarded as a preparatory action for standing, if there is a risk in the subject 11 standing up (in the case of a patient who is likely to fall and the fall leads to a serious accident), before standing up. It is possible to recognize the sitting position, which is a preparatory action, and issue a warning. On the other hand, if the height is 140 cm or more in a place other than the bed 12, it is estimated to be standing. When the head position is moving in the standing position, it can be recognized as a walking state.

Furthermore, the walking speed and walking distance can be calculated from the relationship between time and the amount of movement. The exercise data acquired in this way can be utilized for the health care of the subject 11. In addition, if the head height is 50 cm or less outside the bed 12, there is a possibility of falling. When the height of the head suddenly drops from the standing position outside the bed 12, it is determined that the person has fallen and an alarm can be issued. Further, when the head height is 50 cm or less outside the bed 12 from the lying position, the long sitting position, and the end sitting position on the bed 12, it is determined that the person has fallen from the bed 12 and an alarm can be issued.

By grasping the indoor layout and the spatial position of the head in this way, it is possible to recognize the state (posture and movement) of the subject 11 with high accuracy. If the posture and exercise of the subject 11 in daily life can be recognized, it becomes possible to detect a dangerous state such as an accident of the subject 11 and grasp a long-term change in the health condition. By using the heat-sensitive image pickup device 10 of the present invention, the human body can be clearly identified as a high temperature portion. The bed 12 is shown in the center of the visual field, and the posture of the subject 11 such as lying down, sitting on the edge, standing, walking, and going out can be identified from the positional relationship between the bed 12 and the human body. Although the posture can be recognized with some accuracy only by the position of the head displayed on the image, it is possible to determine the posture with higher accuracy by knowing the coordinates of the head in the space.

Further, it is possible to determine the awakening / sleeping state and the light / deep sleep during sleep from the amount of body movement on the bed 12. It is known that the frequency of appearance of body movement during sleep decreases as the sleep stage becomes deeper. If the number of body movements exceeding the threshold value that occurs in 15 minutes during sleep is defined as the body movement density, the body movements occur continuously during REM sleep or light sleep, and the body movement density increases. However, there are individual differences in this body movement density, and it is difficult to estimate the sleep depth using a uniform threshold value. Therefore, the average value of the body movement density during one week of sleep of the subject 11 is calculated, and light sleep occurs when the body movement density exceeds the average value of the body movement density, and deep when the body movement density is lower than the average value. Determined to be sleep.

FIG. 16 shows the amount of body movement on the bed 12 measured by using the heat-sensitive image pickup device 10 of the present invention, and the sleep depth calculated based on the amount of body movement. It can be seen that the sleep depth can be measured based on the body movement data. Since it is considered that the change in the sleep state of the subject 11 is linked to the health state, it is possible to grasp the change in the health state by using such data and acquiring and analyzing it over a long period of time, and appropriate sleep. It is expected that providing guidance and lifestyle guidance will lead to long-term health maintenance. Long-term sleep irregularities, increased nocturnal arousal, and increased total sleep time are considered signs of cognitive decline. If mild cognitive impairment (MCI) can be detected at an early stage from changes in sleep habits, it is considered to be effective in preventing dementia.

Furthermore, it becomes possible to monitor heartbeat and respiration from changes in head temperature. FIG. 17 shows the data obtained by measuring the facial temperature measured by the heat-sensitive image pickup device 10 and the amount of movement of the chest by respiration measured by the laser positioning meter in a resting state on the bed 12 side by side with the time axis aligned. rice field. Both devices were set to acquire data at 0.1 second intervals to acquire data. From the vertical movement of the chest due to breathing, breathing with a cycle of about 4 seconds can be clearly confirmed. On the other hand, the facial temperature is noisy, and biological signals such as respiration and heartbeat cannot be confirmed. The frequency analysis result of the data (512 data) for 51.2 seconds is shown in FIG. From this result, a peak of 2.7 Hz (cycle 3.7 seconds) caused by respiration and a peak of 1.2 Hz (72 BPM) caused by heartbeat can be clearly observed. In this way, biometric information can be acquired by frequency analysis of long-term data. From the analysis results, it was found that it is necessary to analyze the data for a time of 30 seconds or more for respiration and 50 seconds or more for heartbeat. Furthermore, respiration could be measured from the frequency analysis of minute body movement data of the head. Therefore, by analyzing the change in head temperature during sleep and the amount of head body movement in combination, it has become possible to perform respiratory measurement with higher accuracy.

Compared to the respiratory signal included in the facial temperature data, the heartbeat signal has a high noise ratio and the measurement becomes unstable. In addition, the data shown in the figure was acquired in a stationary state without body movement, so analysis was performed using the data, but it was difficult to measure respiration and heart rate when there was body movement. In principle, it is considered possible to extract biological signals from data accompanied by body movement by body movement correction, but at present, it is difficult to accurately extract biological signals from data containing large noise of body movement. rice field. For this reason, we decided to acquire respiration and heart rate data from sleep data with small body movements in this system.

When a large body movement occurred even during sleep, the data at that time was excluded and not used. It also excludes data in the case of prone position or wearing a futon on the head. Respiratory rate or heart rate data in a stable state can be compared with the average respiratory rate / heart rate during sleep, and an alarm can be issued if there is an abnormal change in respiratory rate / heart rate. ..

As described above, by recognizing and analyzing the temperature image output from the heat-sensitive image pickup device 10, the position, posture, behavior, and emergency situation such as a fall, sleep state, and heart rate of the person to be watched 11 in the room. , Respiratory rate, etc. can now be detected.

Next, the output regarding the current state of the detected target person 11 will be described. Under normal watching conditions, the temperature image shown in the upper part of FIG. 19 is not recorded and output, and the system is set so that the pictogram shown in the lower part of FIG. 19 displays the current state of the subject 11 in real time. Has been done. This is to give priority to the protection of the privacy of the subject 11. However, depending on the condition of the target person 11, the nursing care system in the case of a nursing care facility, and the surrounding support system in the case of an elderly person living alone, it is possible to record and output a temperature image by changing the system settings. It is a system design.

The displayed posture, the detection of the situation, and the types of alarms are summarized in FIG. As the posture, lying position, long sitting position, end sitting position, standing position, walking, and going out are determined. The timing of alarm generation varies depending on the medical condition and situation of the subject. The timing of issuing this alarm is also set when the system is started. In the case of a patient who has a high risk of falling by standing up, an alarm is generated in the sitting position, which is a preparatory movement for standing up. When walking is risky, an alarm is issued while standing, and in the case of patients who are restricted from going out, an alarm is issued when they approach the door or when they go out. It also issues an alarm when a fall or fall is suspected due to changes in posture over time. An alarm can be issued even if there is no movement for a long time or there is very little movement. Even if the body temperature, heart rate, and respiratory rate in normal life are significantly different from the current body temperature, heart rate, and respiratory rate, an alarm is issued along with information on body temperature, heart rate, and respiratory rate. In this way, by analyzing the temperature image, it is possible to grasp the various postures, exercises, and health conditions of the subject and discover abnormalities.

(Example 2)
As a second embodiment of a monitoring monitoring system using a heat-sensitive image pickup device 10 having an optical axis off-axis reflection / refraction type, an example in which deep learning (deep learning or machine learning) is used to detect a posture and a situation. explain. In this embodiment as well, the target application is an elderly watching system installed in the living room of an elderly facility.

In the first embodiment, a method of using a rule-based algorithm for detecting the posture and situation of the subject was described. Deep learning can also be used to detect posture and situation. Hereinafter, a method of performing learning using a temperature image output from the thermal image pickup device 10 by deep learning and determining the posture will be described.

FIG. 21 shows a heat-sensitive image pickup device 10 having a design different from that of FIG. A parabolic mirror 2-1 was used as the convex mirror. By making the distance between the parabolic mirror 2-1 and the far-infrared camera 3 close to each other, the device is miniaturized while maintaining the viewing angle. In the example of FIG. 21, the same single board computer 5 as the apparatus shown in FIG. 1 was used, but the curved mirror was downsized and designed to be vertical to reduce the size. Since the processing capacity of the single board computer 5 alone was insufficient, Movidius manufactured by Intel was connected to the single board computer as an accelerator 5-1 for deep learning to perform image recognition.

If a small computer in which an accelerator 5-1 for deep learning and a single board computer 5 are integrated is adopted, the size of the entire device can be reduced to about 3x3x3 cm. Further, in the design of FIG. 21, a far infrared window 7 is used. As the material of the far-infrared window 7, high-density polyethylene having a high transmittance of far-infrared rays was adopted. Since the high-density polyethylene is white, it is designed so that the far-infrared camera 3 inside cannot be seen. Making the monitoring monitoring device smaller and less noticeable is also effective in reducing the psychological resistance of being monitored by the installed thermal imaging device 10.

For deep learning, Google's framework Tensor Flow was used, and the learning model was MobileNet. Learning was performed on the cloud, the created learning model was installed on the single board computer 5 of the heat-sensitive imaging device 10, and image recognition was performed on the single board computer 5 (edge computing). Since the CPU of the single board computer 5 was insufficient in processing power and could not recognize the image of 5 FPS, the AI accelerator 5-1 was connected to the single board computer 5 to make a judgment.

Of the postures shown in FIG. 20, seven postures (lying position, long sitting position, end sitting position, standing position, sitting position, falling, going out) were determined by using deep learning. First, the temperature images of the above 7 postures are acquired. The temperature image acquisition was performed in the room shown in FIG. 200 images in each posture were continuously acquired at one image per second in order from the recumbent position. Images were acquired while changing the position on the bed 12, the orientation of the body, and the position of the futon little by little while lying in the lying position for 200 seconds. Images of 7 postures were acquired in the same manner by changing the postures in sequence, and 10 sets of images were acquired with this as one set.

The reason why the images were not continuously acquired in the same posture is to avoid being affected by unintended image changes due to changes over time. As a result, the temperature of 2000 sheets in each posture, 14000 sheets in total, was acquired. After attaching label information indicating the posture to all the images, 1000 images of each posture (7000 in total) are used as teacher data images, and a model is created on a notebook PC using the machine learning software library TensorFlow. went.

The model accuracy was confirmed using the remaining 7,000 images that were not used for learning, and the accuracy of 97% was confirmed. Edge computing has made it possible to determine the posture of the subject in real time from the output temperature image of the thermal imaging device 10 using machine learning.

Even when machine learning is used, the posture change is determined by the same method as in the first embodiment. When the posture determination is standing, it is determined whether it is standing (stationary) or walking by comparing it with the previous image. In addition, a comparison is made with the previous posture, and an alarm is issued when a change occurs from a standing position to a fall, a sitting position to a fall, or a recumbent position to a fall. Even when the person changes from the sitting position to the standing position or when going out, an alarm is set according to the preset alarm setting of the target person.

Regarding machine learning, it is known that the quantity and quality of learning data have a strong influence on recognition accuracy. Two approaches can be considered to improve the recognition accuracy. There are two methods, one is to accumulate the amount of data and improve the accuracy while increasing the number of subjects, and the other is to create a learning model for each subject. When a model is created for each individual, the installation location and target person are fixed, so recognition accuracy can be improved even with a relatively small amount of data. Even when this method is used, it is possible to continuously improve the accuracy by acquiring data and recreating the model while continuing to use it. In this case, it is necessary to acquire data for model creation when the system is installed, but since it is possible to create a model for each individual, highly accurate recognition becomes possible.

(Example 3)
About watching surveillance system including video camera

Next, as a third embodiment of a watching monitoring system using a heat-sensitive image pickup device having an off-axis reflection / refraction type optical system, a watching monitoring system including a video camera having sensitivity to visible light and near-infrared light. Will be described. Even systems that include video cameras do not shoot video under normal conditions to ensure privacy. When an abnormality is detected from the temperature image of the thermal image pickup device and an alarm is generated, the video camera is activated and shooting is started. When the room is dark, such as at night, when an alarm is generated, the near-infrared light source installed in the video camera unit is automatically turned on, and a clear image can be captured. The image of the video camera is transmitted to the information terminal when there is a request from the external information terminal. Caregivers (or family members) will be able to use this image to get a clearer picture of the situation. Whether or not the system includes a video camera is selected according to the wishes of the person being watched over and their parents, caregivers, family members, and facility managers.

FIG. 22 shows a flowchart of control of the monitoring monitoring system. The example of FIG. 22 shows a system that includes a visible and near infrared video camera 32. While the monitoring monitoring system is in operation, temperature images are sent from the thermal image pickup device 10 to the single board computer 5 at regular time intervals. The single board computer 5 has functions as an image processing unit 33, a determination unit 34, a data storage unit 35, and an output unit 36. The image processing unit 33 analyzes the two-dimensional infrared radiant energy image captured by the heat-sensitive image pickup device 10 and detects the position of the subject on the two-dimensional image and the infrared radiant energy contained in the image. The determination unit 34 determines the condition of the subject based on the position on the two-dimensional image of the subject and the infrared radiant energy detected by the image processing unit 33. The data storage unit 35 records the status of the target person determined by the determination unit 34. The output unit 36 outputs information indicating the status of the target person obtained by the determination unit 34.
More specifically, the temperature image (input image) from the heat-sensitive imaging device 10 is sent to the image processing unit 33, and the image processing unit 33 calculates the position of the head in the three-dimensional space. The amount of movement is calculated from the comparison between the head position data of the previous frame and the current head position data. The previous frame referred to here may be a plurality of frames including the immediately preceding frame.

Furthermore, the current posture / behavior of the subject is estimated from the position of the head. The determination unit 34 compares the current position and posture with the position and posture of the front frame, and if there is a sudden change, determines whether or not an abnormality such as a fall or a fall has occurred based on the content of the change. Further, the determination unit 34 refers to the data on the average relationship between the position and the body temperature for each hour stored in the data storage unit 35, compares this data with the data of the current position and the body temperature, and abnormally hyperthermia or It is judged to be abnormal at the time of hypothermia. The acquired data is stored in the data storage unit 35. Further, the determination unit 34 determines whether or not there is an abnormality in the heartbeat or the respiration frequency. Data of temperature image, head position information, movement amount, posture / behavior estimation value, sleep state, respiration, and heart rate are stored in the data storage unit 35.

When an abnormality is detected by the determination unit 34, an alarm is transmitted from the output unit 36 to an information terminal such as a tablet or smartphone owned by the caregiver or a computer in the office by wireless communication. At the same time as the alarm, the operation of the video camera 32 is started, and the video image is stored in the data storage unit 35. The information to be transmitted is the time of occurrence of the abnormality, the content of the abnormality, the current visible and near-infrared images, the current temperature image, and the temperature image for several seconds before and after the occurrence of the abnormality.

The caregiver who confirms the alarm confirms the visible and near-infrared images and temperature images displayed on the screen, and the temperature images before and after the occurrence of the abnormality, and confirms and assists as necessary. At this time, if the system includes a call function, a call may be made from the mobile terminal through a speaker to check the status of the target person. Since the caregiver can check the situation of the target person on the spot, it is possible to provide quick and efficient care, improve the quality of care, and reduce the burden on the caregiver.

In the above system operation, video recording is started when an abnormality occurs. However, the user may request a clear image when an abnormality occurs. In order to meet such a demand, instead of starting the video shooting after detecting the abnormality, the video shooting is always performed, and the video image is saved in the data storage unit 35 for a certain period of time (for example, 30 minutes). , The image data that has passed it is deleted, and when an abnormality occurs, the accumulated image data is not deleted and remains. The video camera is always operated, and when an abnormality is detected, the image before and after the abnormality can be saved by saving the data, and the video image at the time of the abnormality can be confirmed from the mobile terminal. .. In this case as well, the video image is not saved in a normal state in consideration of privacy.

(Example 4)
As a fourth embodiment of the watching monitoring system using the thermal imaging device of the present invention, a watching monitoring system for the elderly living alone will be described with reference to FIG. 23. The thermal imaging device 10 is installed in the bedroom of the elderly person 11 who lives alone, who is the person to be watched over. When the watching monitoring system detects an abnormality, a mobile terminal or computer terminal is wirelessly communicated to a guardian 23 such as a family member, a close relative, a watching service provider, or a person in charge of a local government registered as a guardian 23 in advance. An alarm is sent to a terminal 21 such as (not shown). The guardian 23 confirms the terminal display content 22 (warning and temperature image data before and after the abnormality detection) displayed on the terminal 21, and if necessary, rushes, requests an ambulance, and confirms with the residents in the vicinity of the target person. Necessary measures such as request and confirmation by telephone can be taken.

Not only the images before and after the occurrence of the abnormality from the terminal 21 side, but also the current temperature image can be watched and sent from the monitoring system at the request from the terminal 21 side and confirmed on the terminal 21. From the viewpoint of privacy protection, the sending of images is currently limited to the time when an abnormality occurs, and the mechanism is such that image requests from the terminal 21 cannot be made in daily life. At the time of this abnormality confirmation, if the monitoring monitoring system is a system including a video camera, it is possible to confirm the image of the current video camera by selecting the transmission of the video camera image from the terminal 21. .. The monitoring monitoring system can also include devices with remote communication functions such as microphones and speakers. When an abnormality occurs, the guardian 23 can confirm the image on a terminal 21 such as a mobile terminal or a computer terminal, and if necessary, call the watching target person 11 by using the remote call function. If the call is properly answered, measures such as rushing to the living room will not be necessary. In this way, if the safety status of the subject 11 can be confirmed using the terminal 21, the burden on the caregiver and the family can be reduced.

(Example 5)
As a fifth embodiment, a monitoring monitoring system in a nursing care facility, a hospital, or the like will be described. The watching monitoring system has not only a watching function but also a healthcare function. As shown in FIG. 24, the thermal imaging device 10 installed in each room is wirelessly connected to the host computer 20 arranged in the facility, and the data in each room is collected by the host computer 20. In this case as well, the analysis of the temperature image of the thermal image pickup device 10 is performed not by the host computer 20 but by the single board computer 5 built in the thermal image pickup device 10. Under normal conditions, the image is not transferred to the host computer 20, and the image data analyzed by the single board computer 5 is discarded as it is if no abnormality is detected, and is stored in the single board computer 5 and in the host computer 20. Not done. In a normal state, the single board computer 5 sends the position data, posture data, and exercise amount data of the monitoring target person 11 to the host computer 20, and the host computer 20 analyzes the life data (life log) of each target person 11. It is put together.

When an abnormality occurs, the single board computer 5 sends an alarm and images before and after the abnormality occurs. These alarm data are sent to the terminal 21 owned by the guardian 23, and after confirming the alarm content, request an image of the current state as necessary and confirm it on the terminal 21. The guardian 23 confirms the call from the terminal 21 by an intercom or the like or by visiting the room.

In the daily state, the terminal 21 of the caregiving person displays a pictogram showing the state of the target person as shown in FIG. 24 in a form that can list each room. The caregiver can confirm the condition of the care recipient on the terminal 21 and decide the procedure of the caregiving work. For example, the morning caregiving work (breakfast, brushing teeth, changing clothes, etc.) can be carried out in order from the person who is awake, and the caregiving work can be carried out efficiently by putting off the person being assisted during sleep. Furthermore, it becomes possible to wake up (alarm) at a timing when sleep is light. In this way, by providing assistance in accordance with the life rhythm of the person being assisted, it is possible to improve the efficiency of the assistance work and reduce the burden on the person being assisted. By performing various assistance tasks (for example, medication and changing clothes) in accordance with the life rhythm of the care recipient, it is possible to reduce the burden on both the caregiver and the care recipient.

In the above-mentioned system, the temperature image taken in a normal condition is set so as to be deleted after the determination without being saved. In long-term care facilities, there is a desire to save images without erasing them in order to grasp the state of long-term care and to know the situation at the time of an accident or the like. In such a case, it is possible to constantly save the temperature image in the host computer 20 by changing the system setting. Since the temperature image has a lower resolution and a smaller data capacity than the commonly used visible light image, it is less restricted by the capacity of wireless communication, and images of a plurality of rooms can be transmitted and received at the same time and stored in the host computer 20. Is possible. Here, the host computer 20 is installed in the facility, but the role of the host computer 20 can be replaced by cloud computing. Since image recognition is performed by the single board computer 5 built into the monitoring monitoring system placed in each room, it is not necessary to send image data to the cloud in real time, so there are issues with communication capacity and communication delay. Does not occur. By eliminating the host computer 20 by cloud computing, the cost of the system can be reduced.

The monitoring monitoring system of the present invention can be useful not only for the monitoring function for the safety of the care recipient but also for the health care of the care recipient through accurate analysis of the daily life of the care recipient. In facilities such as long-term care facilities and hospitals, care recipients spend most of their lives in private rooms or hospital rooms, so the monitoring and monitoring system of the present invention makes it possible to obtain accurate records of daily life. A wristwatch-type or clothing-type wearable sensor may be used for the purpose of monitoring such living conditions, but it is continued because there is resistance to wearing it all the time and the care recipient removes the sensor. There was a problem that it was difficult to monitor the system. In addition, wearable sensors need to be charged, replaced with batteries, and collected data at high frequency, which increases the burden on caregivers and has not been widely used in nursing care facilities.

In the monitoring monitoring system of the present invention, life can be constantly monitored without being restrained in daily life, and data is automatically sent to the host computer 20, so that both the caregiver and the care recipient bear the burden. It is possible to acquire and analyze living data without having to do so. FIG. 25 shows an example of life analysis. FIG. 25 shows the life pattern during the day. FIG. 26 shows the daily life record of the month in which the data of FIG. 25 is summarized for one month. FIG. 27 shows a monthly life record. FIG. 28 shows the indoor walking distance per day. It can be seen that the amount of exercise decreases from the first half to the second half of the month. From FIG. 26, there is a tendency that the activity decreased in the latter half of the month and the time spent in bed increased.

From the analysis of long-term annual data, it is possible to grasp long-term changes in life. The relationship between sleep state and the progression of dementia and the relationship between daily activity and dementia have been recognized, and it is important to grasp the constant sleep state and motor state for recognition of medical conditions and lifestyle guidance. Similar data analysis is possible for heart rate data and respiration data during sleep. Vital data is useful not only for grasping changes in physical condition and medical condition in the short term, but also for health management of care recipients by acquiring and analyzing long-term trends.

By using the monitoring monitoring system of the present invention in this way, it is possible to analyze not only the monitoring for the safety of the care recipient but also the data for the long-term health care of the care recipient and use it for health promotion. become. Furthermore, by acquiring such daily life data (life log) and analyzing the data together with other health data (big data analysis), data on lifestyles for maintaining and promoting health can be acquired. And analysis becomes possible.

(Example 6)
As a sixth embodiment of the present invention, an example of a watching service will be described. The issue of the monitoring system for safety is the accuracy of abnormality detection. If there are many false positives, the caregiver needs to confirm the safety each time an alarm is issued, which increases the load. The system of the present invention makes it possible to check the situation on a mobile terminal without going to the room and reduce the load, but when there are many false positives, the business is interrupted each time and attention is always paid to the alarm. The psychological burden is great. In particular, in a system in which an alarm reaches the family, the alarm may be issued 24 hours a day, regardless of the time, which increases the burden on the family. On the contrary, if the frequency of alarm generation is reduced in order to reduce false positives, the abnormality may be overlooked and the worst result may be obtained. It is extremely important to improve the accuracy of abnormality detection by improving the accuracy of the monitoring monitoring system. Therefore, in the present invention, a heat-sensitive imaging device capable of directly detecting the human body is adopted. As a result, the recognition accuracy has been improved, but it is still an important issue to suppress false detections and oversights.

In the future, there is a possibility that AI (artificial intelligence) technology will realize higher accuracy, but at present, it is more accurate for humans to check images, and oversight can be suppressed. Therefore, there is a demand for a service that determines the necessity of dealing with an abnormality when the system detects an abnormality. In this service, when an abnormality is detected in a monitoring system installed in a room such as a long-term care facility or in an elderly person living alone, the monitoring service staff does not send an alarm directly to the nursing staff or family members. The data will be sent to. It is a service in which the person in charge of the watching service judges the image and then if the person in charge of the watching service determines that it is necessary, the person in charge of the care service or the family is contacted.

A conceptual diagram is shown in FIG. The watching service staff 23-1 can watch over multiple facilities and houses 24 hours a day. In this case as well, the watching service staff 23-1 does not always watch the image, but when an abnormality occurs, the image before and after the abnormality occurs and the current image are checked, the urgency is judged, and necessary measures are taken. The family member and the guardian 23 do not have to respond to all the alarms generated by the monitoring monitoring system, and only need to be contacted and responded when the monitoring service staff 23-1 deems it necessary. As a result, the burden on the family and the guardian 23 can be reduced. For the guardian 23, dealing with late nights and holidays has a large psychological and physical burden, and it is considered that reducing the burden by such a service is particularly effective.

In the case of the elderly living alone, if the family lives in a remote area, it is often not possible to take immediate action even if an emergency call is received. A rush service has already been provided in case of such a situation. When there is a concern that something is wrong, the service staff 23-2 rushes to the house where he lives alone and checks the situation on behalf of his family. This rush service and the watching service can be linked. If an abnormality is detected by the monitoring monitoring system and the monitoring service determines that the target person needs to be rescued or confirmed, the monitoring service staff 23-1 contacts the rushing staff 23-2, and the rush service staff 23- 2 is a service that rushes to the target person's residence.

Since the monitoring service only needs to check the image when an abnormality is detected by the monitoring monitoring system, it is possible to take charge of many facilities at the same time, and it is possible to respond efficiently and promptly. With the aging of society, highly reliable, efficient, and low-cost monitoring and monitoring systems that utilize IoT (Internet of Things) devices are becoming important. Systems that can efficiently support the safe and secure life of the elderly by utilizing IoT are expected to become more and more important in the future.

So far, we have focused on the monitoring system installed on the bed in the living room, but the installation location is not limited to this. In long-term care facilities and hospitals, it is effective to install a watching monitoring system on the bed because the person to be watched spends most of the time in a private room or a large room in one room with a bed. On the other hand, elderly people living alone spend more time outside the bedroom. In such a case, it is necessary to watch over the place other than the bedroom. Depending on the location, in the living room, a watching monitoring system using a heat-sensitive imaging device 10, and in a toilet, corridor, entrance, etc., by combining with a watching monitoring system using a simple charcoal sensor, etc., it is possible to watch over the entire house. Reliable monitoring is possible. In the monitoring monitoring system using the thermal imaging device 10 used in the living room or the like, it is necessary to change the image recognition software. Here, too, the basic configuration is the same: grasping the three-dimensional position of the subject in the room, recognizing the standing, walking, sitting, and lying positions, and monitoring the occurrence of accidents such as falls and the amount of exercise. ..

We have shown examples of applications for watching over elderly people and inpatients, but the applications are not limited to this. It can also be applied to watching over infants, watching over detainees, watching over pets, and so on. In addition to watching, it can also be used as a surveillance camera to detect humans in streets, stores, and corridors of apartment buildings where ordinary surveillance cameras are used.

Here, application examples other than watching will be described. Higher-performance control is possible by using the heat-sensitive image pickup device 10 of the present invention for applications such as an automatic lighting switch, an automatic door switch, and an intrusion alarm when an inexpensive pyroelectric sensor has been used in the past. become. For example, in the case of a switch that turns on the lamp when it detects entering the toilet, the pyroelectric sensor detects the movement of the person, so if the person continues to be stationary, the lamp will turn off even while in the room. There was a problem.

By using the heat-sensitive image pickup device 10 of the present invention, the existence can be detected even when a person is stationary, so that such a problem does not occur. Further, the number of people in the toilet can be measured, and the usage status of all the private rooms in the toilet can be grasped by one heat-sensitive image pickup device 10. By sending this information by wireless communication, it becomes possible to grasp the usage status of the toilet outside the toilet. This makes it possible to display the vacancy status of the toilet outside the toilet, and to check the vacancy status on the mobile terminal and select the toilet.

In recent years, services such as recognizing vacant seats in stores such as restaurants with a camera and displaying congestion status and vacant seat information outside the store or on the Internet have become widespread. By using the heat-sensitive imaging device 10 of the present invention when acquiring such vacant seat information in a store, a person can be detected from the body temperature, so that the vacant seat can be grasped more accurately. Furthermore, it is possible to grasp the number of visitors and analyze the customer's staying time and the movement route in the store. It can also be installed in shopping malls and the like to analyze changes in the number of customers and customer flow lines.

For use in stores, it is possible to improve recognition accuracy by inputting seat arrangement information as advance information in advance. By using the heat-sensitive image pickup device 10 of the present invention, there is an advantage that there is no problem with privacy such as image leakage as compared with the method using a video camera.

In addition, it can be used as a smart remote controller for home electric appliances by taking advantage of the function of being able to recognize human behavior by image recognition using the heat-sensitive image pickup device 10 of the present invention. There are many recent home appliances that can be connected to Wifi or Bluetooth, and the behavior of the target person is grasped using the monitoring monitoring system of the present invention, and the home appliances are appropriate for the user's life by wireless communication. Can be controlled to.

For example, when the subject falls asleep while watching TV on the bed, it is possible to detect falling asleep, turn off the TV, turn off the lights, and so on.

In addition, various uses are conceivable, such as controlling the air conditioner according to the sleep state to adjust the room temperature, and turning off the power of the lighting, the television, the air conditioner, etc. when the subject goes out of the room.

Although an example of arranging the heat-sensitive image pickup device 10 on the ceiling and using it for watching and monitoring the room has been shown, the installation method of the heat-sensitive image pickup device 10 is not limited to this. It is also possible to invert the positional relationship upside down and install it on the floor or a wall surface near the floor to watch over the room including the ceiling. In monitoring monitoring in an open space without a wall, it is effective to arrange the thermal imaging device 10 on the floor and use it.

(Example 7)
Thermal imaging device for in-vehicle monitoring

Up to this point, examples of an indoor watching and monitoring system for the elderly and infants have been described, but the use of the watching and monitoring system of the present invention is not limited to this, and is applied to various other uses. Is possible. Here, as a seventh embodiment, application to a watching monitoring system that is mounted on an automobile to detect a pedestrian will be described.

The required viewing area differs between the in-vehicle monitoring system and the indoor monitoring system. An in-vehicle system for detecting a pedestrian requires a wide field of view in the horizontal direction. On the other hand, a wide field of view is not required in the vertical direction. In order to prevent entanglement when turning left or right, a field of view of 90 degrees (viewing angle 180 degrees) or more is required for each of the left and right sides with respect to the traveling direction of the vehicle. On the other hand, in the vertical direction, a viewing angle of about 30-40 is sufficient. Therefore, it is required to have different viewing angles in the horizontal direction and the vertical direction.

As already mentioned, the conventional far-infrared camera has a narrow viewing angle of 40-50 °, and its performance is insufficient as a pedestrian detection system. Since the catadioptric system using the convex mirror 2 of the indoor watching monitoring system described so far is designed to widen the viewing angle in both the horizontal direction and the vertical direction, the pixels in the vertical direction cannot be effectively used.

Therefore, we have developed an optical system that can widen the viewing angle only in the horizontal direction for in-vehicle use. As already mentioned, the field of view can be expanded by using a convex mirror. By changing the mirror shape in the horizontal direction and the vertical direction, it is possible to control the viewing angle in the vertical direction and the horizontal direction.

Here, the optical system of the thermal imaging apparatus 10 in which the viewing angle in the horizontal direction is widened by using a cylindrical mirror will be described with reference to FIG. In the optical system shown in FIG. 30, a cylindrical mirror 2-2 is arranged as a convex mirror at an angle of 30 ° with respect to the vertical direction. When the optical axis 3-4 of the cylindrical mirror 2-2 is defined in the outer peripheral direction of the circle, the angle α formed by the optical axis 3-4 of the cylindrical mirror 2-2 and the optical axis 3-3 of the far-infrared camera 3 is defined. The far-infrared camera 3 was arranged off-axis so that α = 30 °. Since the curvature of the cylindrical mirror 2-2 in the tubular direction (vertical direction) is zero, the viewing angle in the vertical direction is the vertical viewing angle of the far-infrared camera 3 as shown by the typical ray 6. It will be reflected as it is. Here, a stainless steel cylindrical mirror was used as the cylindrical mirror 2-2. As the far-infrared camera 3, a bolometer-type camera with a viewing angle of 56x42 ° and 160x120 pixels was adopted.

FIG. 31 shows a view of the cylindrical mirror 2-2 as viewed from above the cylinder. FIG. 31 shows light rays for each viewing angle of 5 ° of the far-infrared camera 3 and light rays reflected by the circular mirror. As can be seen from the light rays reflected by the circular mirror of FIG. 31, it can be seen that the viewing angle is expanded and a wide viewing angle of 180 ° or more is obtained.

It can be seen that by using the cylindrical mirror 2-2, the horizontal field of view can be expanded while maintaining the vertical field of view. The thermal imaging device 10 is such that the far-infrared camera 3 is arranged off-axis with respect to the cylindrical mirror 2-2 so that the far-infrared camera 3 is not reflected in the center of the image.

In FIG. 32, the imaging range of the heat-sensitive image pickup device of the present invention when the heat-sensitive image pickup device 10 for vehicle is attached to the front portion of the roof of the automobile 27 is compared with the case where a normal far-infrared camera is used. Indicated. The area shown by the broken line represents the field of view of a normal far-infrared camera, and the area shown by a solid line represents the field of view when the far-infrared imaging apparatus 10 of FIG. 30 is used. A far wider viewing angle of 220 ° is obtained as compared with the viewing angle of 55 ° of a normal far-infrared camera.

FIG. 33 shows the field of view shown in FIG. 32 with respect to the road. The broken line represents the field of view of a normal far-infrared camera, and the solid line represents the field of view when the thermal imaging apparatus 10 of the present invention is used. With a viewing angle of 55 ° of a normal far-infrared camera, it is difficult to detect pedestrians 28 and bicycles 29 near the automobile 27, and it is not possible to detect pedestrians getting caught when turning left or right or jumping out of an intersection with poor visibility. You can see that. Since these accidents account for most of the accidents resulting in injury or death in urban areas, it is particularly important to detect pedestrians 28 and bicycles 29 in the vicinity of the automobile 27.

It can be seen that the vehicle-mounted heat-sensitive image pickup device 10 of the present invention has a wide viewing angle of 180 ° or more (220 ° in the example of FIG. 33) and can detect pedestrians, motorcycles, and bicycles in the vicinity of the automobile.

A far-infrared image of a person on an actual road is shown in FIG. The vehicle is in the left lane of the two lanes, and the figure shows the center line, the ends of the left and right lanes, and the distance from the vehicle to the pedestrian. Since the car is in the left lane, the left lane is shot in the center of the screen. The distance between the pedestrians on the left and right sidewalks and the car is 50, 20, 10, 5, 2, 0 m. 0 m indicates that there is a pedestrian right next to the thermal image pickup device 10 installed in the vehicle.

Since the cylindrical mirror 2-2 is used, the distortion of the image is large, but it can be seen that the images are taken from a pedestrian at a distance of 20 m from the vehicle to a pedestrian at 0 m (right beside the vehicle). Detection of a pedestrian at a distance of 50 m is difficult from the image of FIG. 34. This is because the resolution of the far-infrared camera used is as low as 160x120 pixels.

If a high-resolution far-infrared camera is used, it is possible to detect a pedestrian at a distance of 100 m or more. However, since a high-resolution far-infrared camera is expensive, it is necessary to select a camera according to the purpose. Here, since the detection of pedestrians in the vicinity of the automobile is prioritized over the detection of pedestrians in the distance, a low-priced low-pixel far-infrared camera was selected.

From the image of FIG. 34, a pedestrian is detected by image recognition, and the driver is alerted as necessary. As already mentioned, the far-infrared camera can detect a person directly from the body temperature, so that the person can be easily detected. Similar to image recognition by the elderly watching and monitoring device, it is possible to identify a person with higher accuracy by the temperature, shape and size on the image.

When a person is detected around the vehicle, an alarm / warning is given to the driver according to the degree of danger. Warnings to the driver are given by warning sounds, warning lights, and displays installed in the vehicle. For example, the volume and pitch of the alarm sound can be changed according to the degree of danger, or the blinking speed of the alarm light can be changed to give a warning. When using a display, a temperature image can be displayed and a person mark can be displayed on the display to notify the warning.

So far, we have described the detection of pedestrians, but since pets and wild animals on the street also have body temperature, they can be detected from temperature images. It is also possible to distinguish between humans and animals by image recognition and issue different alarms.

In the above embodiment, a cylindrical mirror 2-2 having a circular cross-sectional shape in the horizontal direction and a straight cross-sectional shape in the vertical direction is used as the reflection mirror, but the present invention is not limited to this. For example, the cross-sectional shape in the horizontal direction may be an ellipse, a parabola, or a hyperbola. Further, the cross-sectional shape in the vertical direction can be reduced in the viewing angle in the vertical direction and the magnification can be increased by using a concave mirror. By using the concave mirror, the detection sensitivity of a distant pedestrian can be increased by increasing the magnification at the center of the screen. The concave shape of the lens at this time can also be selected from a circular shape, an ellipse, a parabola, a hyperbolic shape, and the like.

FIG. 35 shows an example in which the saddle-shaped mirror 2-3 is adopted. Here, as the saddle-shaped mirror 2-3, a quadric surface called a hyperbolic paraboloid was adopted. A hyperbolic paraboloid is a curve that satisfies the relationship z = (x / a) ^2- (y / b) ^2. The cross section in the horizontal direction is a convex mirror, and the viewing angle is enlarged as shown in FIG. On the other hand, the vertical direction is a concave mirror, and the viewing angle is reduced.

A narrow viewing angle means that an image can be captured at a high magnification, which means that the detection performance of a distant pedestrian is improved. By designing a more complicated three-dimensional shape, the magnification can be controlled by the position on the screen. For example, by increasing the curvature of the concave mirror in the vertical direction at the center of the screen and decreasing it toward the outer circumference, the magnification at the center of the screen can be increased.

Since the thermal image pickup device 10 is installed in the direction of travel of the automobile, the road in the direction of travel is photographed in the center of the screen. As shown in FIG. 34, a distant pedestrian is always shown in the center of the screen. By increasing the curvature of the concave mirror in the vertical direction at the center of the screen, it is possible to image a distant pedestrian with high sensitivity.

As already mentioned, the cross-sectional shape of the convex mirror in the horizontal direction can be increased in magnification toward the periphery of the screen with respect to the center of the screen by decreasing the curvature from a circle to a parabola, a hyperbola, and the outer circumference. .. In the case of in-vehicle use, the pedestrian in the vicinity of the screen is close to the vehicle, so that the pedestrian is photographed sufficiently large as shown in FIG. 34. In the case of in-vehicle use, it is easier to detect a pedestrian in the distance if the cross-sectional shape of the convex mirror in the horizontal direction is a shape in which the curvature increases toward the outer peripheral portion.

In this embodiment, in order to give priority to the detection of pedestrians in the vicinity of the vehicle, a cylindrical mirror 2-2 having a circular cross-sectional shape in the horizontal direction of the convex mirror is adopted.

By controlling the surface shape of the mirror, a desired viewing angle can be designed. Although it is costly to manufacture a mirror having a complicated surface shape, it is possible to manufacture a mirror having an arbitrary shape at a relatively low cost by depositing a metal film on a plastic material that has been injection-molded. Further, since the far-infrared light wavelength is 10 times or more the visible light wavelength, the requirement for mirror processing accuracy can be relaxed by an order of magnitude or more. Another advantage of using a far-infrared catadioptric system is that the manufacturing cost of the mirror can be reduced.

Adopting a high-pixel far-infrared camera to improve the detection performance of pedestrians in the distance will lead to a significant cost increase. On the other hand, changing the mirror shape can be realized at a relatively low cost. The shape of the reflection mirror can be designed according to the performance required for the in-vehicle monitoring device. Further, although the heat-sensitive image pickup device 10 is installed toward the front in the traveling direction here, it is also possible to install the heat-sensitive imaging device 10 toward the opposite side in the traveling direction to detect a pedestrian behind. ..

(Example 8)
Thermal imaging device for driver / passenger monitoring

In the seventh embodiment, an embodiment attached to the front outside of the vehicle to detect a pedestrian on the road is shown. As an eighth embodiment, an embodiment of a thermal imaging device that can be mounted toward the inside of a vehicle to monitor a driver or a passenger will be described.

In recent years, automobiles equipped with a driver monitoring function have been put on the market. It is a function that takes a picture of the driver using a camera and performs personal authentication of the driver, an alarm when driving aside, a drowsiness, an alarm when falling asleep, etc. from the image. By personal authentication with a camera, it is possible to automatically switch seat position, door mirror position adjustment, air conditioner setting, etc. It is common to detect aside driving by identifying the direction of the face, and to take a nap by measuring the degree of eye opening (eye opening degree).

In addition, the development of a system that can monitor not only the driver but also the passengers at the same time is underway. It is a system that can monitor not only the driver but also the passengers with one camera using a wide-angle camera. The passenger's face recognition function enables the passenger to be identified and selected for his / her favorite music, and to operate the in-vehicle device that recognizes hand gestures such as the movement and shape of the passenger's hand.

The problem of driver / passenger monitoring using a conventional camera is the change in the brightness inside the vehicle. The lighting environment inside the car changes drastically from a dark situation at night to a bright state in direct sunlight. In particular, when a part of the inside of the vehicle is exposed to direct sunlight, a large difference in brightness occurs in the image, which makes it difficult for the camera to recognize it.

In order to avoid this problem, it is common to use near-infrared lighting and a near-infrared camera in the in-vehicle monitoring system. However, since the sunlight also contains near-infrared light, there is a problem that the difference between light and dark becomes too large and recognition becomes difficult under the condition of being exposed to direct sunlight.

This problem can be solved by using a far-infrared camera. However, as I have repeatedly stated, conventional far-infrared cameras have a narrow viewing angle, and it was not possible to capture the driver and all passengers with a single camera. By using the thermal imaging device of the present invention, a wide field of view has been realized and all passengers can be monitored.

FIG. 36 shows an installation example of the thermal imaging device 10. The thermal image pickup device 10 is installed in the vicinity of the windshield 25 on the rearview mirror 26 of the automobile toward the room. Here, too, the cylindrical mirror 2-2 is used to widen the viewing angle in the horizontal direction. The thermal image pickup device 10 is attached to the roof 24 portion of the automobile, and the cylindrical mirror 2-2 is vertically attached so that the field of view looks down.

FIG. 37 shows a temperature image of the inside of the vehicle taken by the thermal image pickup device 10 of FIG. 36. The image is reduced in half in the vertical direction to make it easier to see. It can be seen that the driver 30 and three passengers 31 (two passengers in the passenger seat and the back seat) are photographed. Since the person is detected from the body temperature, only the passenger is photographed, not the person other than the person.

From the image, it can be seen that the driver 30 is wearing glasses. Since the temperature of the glasses is lower than the body temperature, they appear black. It can also be recognized from the image that the driver 30 is raising the index finger of his left hand. From the direction of the driver 30's face, it is possible to prevent the driver from looking away and to operate the audio device or the like by recognizing the shape and movement of the passenger's hand. Even from low-resolution temperature images, personal authentication was possible by utilizing face recognition technology based on machine learning.

By using the heat-sensitive image pickup device 10 of the present invention for in-vehicle monitoring, it has become possible to monitor not only the driver 30 but also the passenger 31. As a result, accidents can be reduced by monitoring the degree of attention and drowsiness of the driver 30. Furthermore, the position and movement of not only the driver 30 but also the occupant are monitored, music suitable for the occupant is selected using the occupant's face recognition function, and the movement and shape of the occupant's hand are recognized by hand gesture recognition. It becomes possible to operate the in-vehicle device. Further, in the thermal imaging device 10, it is possible to control the optimum temperature and air volume of the air conditioner for each passenger from the body surface temperature of the passenger. By enabling fine-tuned control for each passenger, comfort during boarding can be improved. In addition, when a child is placed on the back seat, the behavior and state of the child can be grasped from the driver's seat.

Further, there is another use for driver monitoring using the thermal imaging device 10 of the present invention. It is an evaluation of driving stress. It is known that driving stress can be evaluated by measuring the temperature change of the nose, and research is underway (for example, Yamakoshi et al., "A trial of driving stress evaluation using differential facial skin radiation temperature". -Basic study by monotonous operation stress load ", Biomedical Engineering, 48 (2), pp.163, 2010). It is known that driving performance deteriorates when driving stress is too high or too low, and it is important to understand the stress state during driving in order to realize safe driving.

The stress of the driver was measured by using the thermal image pickup device 10 shown in FIG. Stress causes contraction of peripheral blood vessels, which lowers the skin temperature in the peripheral area. Here, the temperature difference between the forehead surface temperature, which has a strong correlation with the trunk temperature, and the nose temperature, which represents the skin temperature of the peripheral part, was measured to evaluate the driving stress. The driver's forehead and nose areas were recognized by image recognition of the temperature image, and the temperature difference was calculated from the temperature of each part. The facial temperature was evaluated by the thermal image pickup device 10 while running for 60 minutes in the drive simulator. It was confirmed that the nasal temperature gradually decreased by performing monotonous operation. Using the temperature difference between the forehead temperature and the nose temperature as an index, it has become possible to evaluate stress and warn the driver.

(Example 9)
Thermal imaging device for autonomous vehicles

In recent years, the development of self-driving cars has been promoted. Not only automobiles, but also various autonomous vehicles that are not operated by humans are being developed and partially put into practical use. For example, autonomous driving heavy construction machines, transport robots, delivery robots, catering robots, and the like have been put into practical use. Large commercial cleaning robots are also commercially available. Unlike factories, autonomous vehicles and robots that operate in an environment where people and pets coexist place importance on safety, especially prevention of contact accidents with people. It is required to automatically recognize people and pets in real time while driving and to drive safely.

The heat-sensitive image pickup device 10 of the present invention can be used for such an application. Since people and pets can be selectively detected from body temperature even in an environment where various objects exist, it is possible to drive safely at a distance from people and pets. Unlike obstacles that do not move, people and pets can move suddenly and have a large impact in the event of an accident. Driving with a drop is especially important for ensuring safety. By using the heat-sensitive image pickup device 10 of the present invention, people and pets can be reliably recognized in a wide range in the traveling direction, so that safe autonomous driving can be realized.

FIG. 38 shows an application example to an autonomous driving vehicle for delivery. The thermal image pickup device 10 is mounted on the roof in front of the vehicle. Other sensors such as a normal camera and a proximity sensor can be arranged in the vicinity of the thermal image pickup device 10.

The heat-sensitive image pickup device 10 recognizes people and pets existing in the traveling direction in real time, and the arithmetic unit connected to the device makes it possible to drive safely while changing the travel route.

The monitoring monitoring system using the thermal imaging device, the thermal imaging device, and the monitoring monitoring method using the thermal imaging device of the present invention are used for monitoring and healthcare of elderly people, pedestrian detection of vehicles, and passengers. Useful for monitoring applications.

1 frame
2 Convex mirror
2-1 Spherical mirror
2-2 Cylindrical mirror
2-3 Saddle-shaped mirror
3 Far infrared camera
3-1 lens,
3-2 Far infrared array sensor
3-3 Exit pupil center point
4 Wiring
5 single board computer
5-1 Accelerator
6 Typical rays
6-1 Rays in the normal direction of the convex mirror
6-2 Rays in the tangential direction of the convex mirror
7 Far infrared window
10 Thermal imaging device
11 Persons to be watched over
12 beds
13 door
14 shelves
15 windows
16 floors
17 desk
18 chairs
19 temperature image
20 host computer
21 Mobile terminal
22 Mobile terminal display contents
23 Parents
23-1 Watching service staff
23-2 Rush service personnel
24 roof
25 windshield
26 Room mirror
27 car
28 Pedestrian

29 bike
30 driver
31 Passenger
32 camcorder
33 Image processing unit
34 Judgment unit
35 Data storage
36 Output section

Claims (2)

Remove the optical axis have a catadioptric system, a thermal imaging device used is placed on a wall of a room,
With an arithmetic unit,
The off-axis catadioptric system includes a far-infrared camera for capturing a temperature image used for detecting a person, a convex mirror having an optical axis inclined with respect to the optical axis of the far-infrared camera, and a convex mirror. Have,
The viewing angle of the off-axis reflection / refraction optical system is 140 ° or more.
The temperature image taken by the far-infrared camera is a highly distorted temperature image utilizing the reflection of the convex mirror.
The arithmetic device recognizes the head of the subject from the size and temperature of the high temperature region in the temperature image, obtains the coordinates of the head on the temperature image, and obtains the coordinates, the temperature, and the size of the head. From this, a watching monitoring system characterized in that the position of the target person in the three-dimensional space is calculated. A thermal imaging device with an off-axis reflection / refraction optical system,
Arithmetic logic unit and
With a car ,
The off-axis catadioptric system includes a far-infrared camera for capturing a temperature image used for detecting a person, a convex mirror having an optical axis inclined with respect to the optical axis of the far-infrared camera, and a convex mirror. Have,
The viewing angle of the off-axis reflection / refraction optical system is 140 ° or more.
The arithmetic unit detects a high temperature region in the temperature image as a person and detects it as a person.
The thermal image pickup device is mounted on the automobile and is mounted on the automobile.
The convex mirror is mounted so as to face the traveling direction of the automobile.
The convex mirror has a convex curved surface shape in the horizontal direction and a linear or concave curved surface shape in the vertical direction.
The far-infrared camera is arranged at an angle with respect to the mirror surface in the vertical direction of the convex mirror.
By increasing the curvature of the cross-sectional shape of the convex mirror in the horizontal direction toward the outer peripheral portion, the magnification of the central portion of the temperature image is higher than that of the outer peripheral portion as compared with the case of using the cylindrical mirror. A watching and monitoring system that features this.

Images