JP2018013929A - Wake-up monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a wake-up monitoring device that is light in calculation processing as light as it can excellently operate on a low resource device, causes light traffic on a network, and can be constructed at low cost.SOLUTION: A wake-up monitoring device 104 is placed at a position where it can photograph an elderly person's face in a state where the elderly person, a subject, wakes up. When the elderly person wakes up, the device detects a face direction vector and a line of sight direction vector of the elderly person from an image data stream output from the imaging device, and transmits an email and/or SNS message of a warning message to a smart phone of an employee in charge of monitoring if it determines that the elderly person is opening their eyes.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a wake-up monitoring device particularly suitable for preventing wrinkles of elderly people.

  It is a well-known fact that Japan is facing an aging society. In addition, there are many incidents involving elderly people in the middle of the night and involved in traffic accidents, which is a social problem. Against this background, the term “watching service” is used as a service for monitoring the elderly and protecting them when necessary, and various elderly monitoring systems have been proposed for this purpose.

The technical contents that are considered to be partially related to the present invention are listed as prior art documents.
Patent Document 1 discloses a wrinkle prevention system that effectively protects a person to be protected who has a risk of wandering using a face authentication collation apparatus.

Japanese Patent Laid-Open No. 2006-110308

The technique disclosed in Patent Document 1 described above needs to register the face of an elderly person to be monitored in advance in the system. In addition, with this technique, the computation processing required for face authentication becomes enormous, which is a factor in increasing the cost of the entire system.
Therefore, it is expected that a system using a low-resource device such as a microcomputer for embedded applications will be used in the future “monitoring service”. To realize this, it will be realized even with hardware that has poor computing power. There is a need for an elderly monitoring system that uses software that has a light processing load.

  The present invention has been made in view of such problems, and the object of the present invention is to realize a low-cost low-resource device that is light in arithmetic processing and can be realized as a “watching service” when an elderly person goes to bed. The object is to provide a wake-up monitoring device that can be used.

  In order to solve the above-described problems, the wake-up monitoring device of the present invention includes an imaging device capable of capturing a face of a subject illuminated by near infrared rays, and an image data stream output from the imaging device. A face detection processing unit that detects presence and outputs face-extracted image data obtained by extracting the face of the subject, and coordinate information in the two-dimensional space including the contour of the face of the subject from the face-extracted image data. And a feature point extraction unit that outputs feature point data that is a collection of feature points. Furthermore, a vector analysis unit that generates a face direction vector indicating the direction of the face of the subject and a line-of-sight direction vector indicating the direction of the line of sight on the face of the subject from the feature point data, the face direction vector and the direction of the line of sight When the vector is both valid, the eye-meditation determination unit that determines that the subject is in an open state and the eye-meditation determination unit determines that the subject is in an open state In response to the output result, a warning message is generated and transmitted to a predetermined device.

According to the present invention, there is provided a wake-up monitoring device that can be implemented with a low-resource device that is light in arithmetic processing and low in cost, and that can be used as a “watching service” when an elderly person goes to bed. Can do.
Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is the schematic which shows the embodiment of the wake-up monitoring apparatus which concerns on embodiment of this invention. It is the schematic which shows the whole image of an elderly person monitoring system including the wake-up monitoring apparatus which concerns on embodiment of this invention. It is a block diagram which shows the hardware constitutions of the wake-up monitoring apparatus. It is a block diagram which shows the software function of the wake-up monitoring apparatus. Schematic diagram showing an example of an image data stream output from the imaging device, schematic diagram showing an example of face extraction image data output by the face detection processing unit, and an example of feature point data output by the feature point extraction unit FIG. It is a figure which shows the example of the feature point data in the state in which the subject has opened eyes, and the example of the feature point data in the state in which the subject has closed eyes.

The applicant has developed a concentration processing system that takes a user who views content with a camera and calculates a concentration based on the direction of the user's face and line of sight (Japanese Patent Application No. 2006-124611).
The concentration processing system captures the user's content by photographing the user's face with a camera, detecting the orientation of the user's face and line of sight, and measuring how much the orientation is directed to the display that displays the content. The degree of concentration with respect to is calculated. At this time, the concentration processing system generates the user's face direction and the user's line-of-sight direction as vector data from the user's face image captured by the camera.

  The inventor paid attention to the fact that the above-mentioned concentration processing system filed earlier can be applied to a monitoring system for elderly people. That is, for the bed or bedding on which the elderly person who is to be monitored sleeps, the above-mentioned concentration is performed at a position where the elderly person's face cannot be seen when the elderly person sleeps and the elderly person's face is visible when the elderly person gets up Deploy degree processing system. Then, this concentration processing system plays a role as an elderly person's wake-up monitoring device. This wake-up monitoring device can also be called a reduced function version of the concentration processing system, and a bed or a futon on which an elderly person goes to sleep is arranged from the direction of the elderly person's feet toward the elderly person.

  In this state, if the near infrared LED lighting device illuminates the room and vector data indicating the face direction of the elderly person is obtained, it can be confirmed whether or not the elderly person has woken up. Furthermore, if vector data indicating the direction of gaze of an elderly person is obtained, it can be estimated that the elderly person is in an open state.

[Installation operation example]
FIG. 1 is a schematic view showing a state in which a wake-up monitoring device 104 according to an embodiment of the present invention is installed and operated in a predetermined facility.
An elderly person 102 goes to bed in a predetermined room 103 at an elderly person facility 101 such as a nursing home or a hospital. In this room 103, a plurality of wake-up monitoring devices 104 according to the embodiment of the present invention are installed. The wake-up monitoring device 104 includes an imaging device 308, which will be described later with reference to FIG. 3, and a wake-up determination function, which will be described later with reference to FIG.
In addition, a near infrared LED illumination device 105 is installed on the wall of the room 103. When the elderly people 102 go to sleep, when the illumination of the room 103 is turned off, the near infrared LED illumination device 105 is turned on and goes to sleep. Illuminate the elderly 102. Needless to say, near-infrared light cannot be seen with the naked eye, so it does not prevent the elderly 102 from going to bed.

  The wake-up monitoring device 104 is installed on the wall of the room 103 and is directed almost to the side. In the state where the elderly person 102 goes to bed, the wake-up monitoring device 104 cannot capture the face of the elderly person 102 who is the subject. When the elderly person 102 gets up from the bed, the wake-up monitoring device 104 can capture the face of the elderly person 102.

  The wake-up monitoring device 104 according to the embodiment of the present invention is significantly different from the existing monitoring camera 107, and in normal settings, it generates moving image data or transmits moving image data to a predetermined host. No action is taken. When the elderly people 102 to be monitored are sleeping quietly, no warning message, information such as an image is transmitted to the outside. When the elderly person 102 gets up, the wake-up monitoring device 104 detects this, generates a warning message, and transmits an e-mail containing the warning message.

  When the wake-up monitoring device 104 captures the face of the elderly person 102 who gets up, an e-mail including a “warning” message indicating that the elderly person 102 to be monitored may have woken up is transmitted through the wireless LAN router 106 or the like. Send to a mail server connected to the Internet. The e-mail is received by the smartphone 110 of the staff member 109 packed in the nurse station 108. When the smartphone 110 receives the e-mail, the staff 109 carrying the smartphone 110 can know that the elderly person 102 is likely to wake up and hesitate.

  When the wake-up monitoring device 104 detects that the elderly person 102 has woken up and creates an e-mail including a warning message, the wake-up monitoring apparatus 104 photographs the elderly person 102 who has woken up and attaches the still image data to the e-mail. The staff member 109 can check which elderly person 102 got up by browsing the image attached to the e-mail on the display of the smartphone 110. In addition, the staff 109 can detect the elderly person's waking by the wake-up monitoring device 104 before displaying the figure of the elderly person who the ordinary monitoring camera 107 wrinkles on the monitor 111. The possibility of oversight can be greatly reduced.

[overall structure]
FIG. 2 is a diagram showing a schematic configuration of the wake-up monitoring device 104 according to the embodiment of the present invention.
One or more wake-up monitoring devices 104 are in an IP reachable state by the wireless LAN router 106 or the like. That is, the wake-up monitoring device 104 is connected to the Internet 201.
When the wake-up monitoring device 104 captures the face of the elderly person 102 who gets up, an e-mail including a “warning” message indicating that the elderly person 102 to be monitored may have woken up is transmitted through the wireless LAN router 106 or the like. Transmission is performed toward the mail server 202 connected to the Internet 201. For this purpose, the wake-up monitoring device 104 stores the mail address of the staff member 109 and the FQDN (Full Qualified Domain Name) of the mail server 202. The staff member 109 recognizes that the elderly person 102 is likely to wake up by receiving an e-mail from the mail server 202 by the smartphone 110.

  In addition, a personal computer having a simple web server function is connected to the wireless LAN router 106 or the like to which the wake-up monitoring device 104 is connected, and a warning message is displayed on the personal computer display instead of or together with the electronic mail. May be. Further, a P2P (Peer to Peer) application may be installed in each of the wake-up monitoring device 104 and the smartphone 110, and the wake-up monitoring device 104 may directly transmit a message to the smartphone 110. That is, the target to which the wake-up monitoring device 104 transmits a message is not limited to the server.

[Wake-up monitoring device 104: hardware configuration]
FIG. 3 is a block diagram showing a hardware configuration of the wake-up monitoring device 104 according to the embodiment of the present invention.
The wake-up monitoring device 104 is a general microcomputer, and a wireless LAN interface 305 and a NIC (Network Interface Card) 306 for connecting to a CPU 301, a ROM 302, a RAM 303, a nonvolatile storage 304, a wireless LAN router 106, and the like are connected to a bus 307. Has been.
An imaging device 308 having an important role in the wake-up monitoring device 104 is also connected to the bus 307. The imaging device 308 is a commercially available CMOS camera, for example. Since some CMOS cameras can detect near-infrared rays, such a commercially available CMOS camera can be used as it is in the wake-up monitoring device 104 of this embodiment.
The nonvolatile storage 304 stores a network OS including a TCP / IP protocol stack for connecting the microcomputer to the network and a program for operating as the wake-up monitoring device 104.

[Wake-up monitoring device 104: Software function]
FIG. 4 is a block diagram illustrating software functions of the wakeup monitoring device 104 according to the embodiment of the present invention.
The image data stream output from the imaging device 308 is supplied to the face detection processing unit 401.
The face detection processing unit 401 regards the image data stream output from the imaging device 308 as a still image continuous on the time axis, and performs, for example, the Viola-Jones method on each image data of the still image continuous on the time axis. The presence of the face of the elderly person 102 is detected using a known algorithm such as the above. And the face extraction image data which extracted only the face of elderly person 102 is output.
The face extraction image data output by the face detection processing unit 401 is supplied to the feature point extraction unit 402.
The feature point extraction unit 402 performs processing such as polygon analysis on the face image of the elderly person 102 included in the face extraction image data. Then, feature point data including the entire face of the elderly person 102, the outlines of the eyebrows, eyes, nose, mouth, and the like and the feature points of the face indicating the pupil is generated. Details of the feature point data will be described later with reference to FIG.

The feature point data output from the feature point extraction unit 402 is supplied to the vector analysis unit 403.
The vector analysis unit 403 obtains a vector indicating the orientation of the face of the elderly person 102 (hereinafter referred to as “face direction vector”) from the feature point data based on the two consecutive face extraction image data, and the line of sight of the face of the elderly person 102. A vector indicating the direction of the image (hereinafter referred to as a “gaze direction vector”) is generated.
In the case of realizing the concentration processing system developed by the applicant, the face direction vector and the line-of-sight direction vector are added after this, and the display for the elderly person 102 to display the content and the imaging device 308 are finally provided. The gaze direction vector that indicates where in the three-dimensional space including the gaze is calculated is calculated, and finally it is determined whether or not the gaze direction of the user is facing the display. Vector addition processing is not performed.

The face direction vector and the line-of-sight direction vector output from the vector analysis unit 403 are supplied to the eye meditation determination unit 404.
The eye meditation determination unit 404 detects the presence / absence of the face direction vector output from the vector analysis unit 403 and the presence / absence of the line-of-sight vector, and creates warning information. The warning information is “Safety” if there is no face direction vector, “Warning” if there is a face direction vector but no gaze vector, and “Danger” if both the face direction vector and the gaze vector are present. ) ”. The fact that the line-of-sight vector is not obtained means that the eyes are closed. Here, it is necessary to be careful, but it is determined that it is not “dangerous”.

The warning information output from the eye meditation determination unit 404 is supplied to the warning transmission unit 405.
When the warning transmission unit 405 receives the warning information “CAUTION” or “DANGER” from the eye meditation determination unit 404, the warning transmission unit 405 acquires still image data at the time of receiving the warning information from the image data stream output from the imaging device. .
On the other hand, the warning transmission information 406 includes a protocol for sending a warning (SMTP (Simple Mail Transfer Protocol) and / or HTTP (Hypertext Transfer Protocol)), and a warning transmission target if the warning is an e-mail. Stores information such as the mail address, FQDN of the mail server 202, and the FQDN or IP address of the web server that provides the SNS if the warning is a web message such as SNS (social networking service). Has been.

  The warning transmission unit 405 refers to the warning transmission information 406, creates a warning message based on the warning information, and outputs the warning message according to the protocol specified in the warning transmission information 406. That is, if the protocol specified in the warning transmission information 406 is electronic mail (SMTP), the warning transmission unit 405 operates as an electronic mail client (MUA: Mail User Agent). Similarly, if the protocol specified in the warning transmission information 406 is web (HTTP), the warning transmission unit 405 operates as a web client. When the warning transmission unit 405 is a web client, it is necessary to follow a protocol uniquely defined by the SNS to which the SNS message is posted, and the procedure for that is implemented in the warning transmission information 406 or the warning transmission unit 405. There is a need to.

The warning message output by the warning transmission unit 405 is transmitted through the interface selection unit 407 to a predetermined mail server 202 and / or web server connected to the network of the wireless LAN interface 305 or the NIC 306.
The entity of the interface selection unit 407 is a TCP / IP protocol stack and a DHCP (Dynamic Host Configuration Protocol) client provided in the OS. That is, the interface selection unit 407 selects a network interface connected to the IP reachable network, and transmits a warning message to the mail server 202 and / or the web server.

[About feature point data]
Hereinafter, operations of the face detection processing unit 401, the feature point extraction unit 402, and the vector analysis unit 403 will be described.
FIG. 5A is a schematic diagram illustrating an example of an image data stream output from the imaging device 308. FIG. 5B is a schematic diagram illustrating an example of face extraction image data output by the face detection processing unit 401. FIG. 5C is a schematic diagram illustrating an example of feature point data output by the feature point extraction unit 402.
First, an image data stream including a subject is output from the imaging device 308 in real time. This is the image data P501 in FIG. 5A.

Next, the face detection processing unit 401 detects the presence of the face of the subject from the image data P501 output from the imaging device 308 using a known algorithm such as the Viola-Jones method. Then, face extraction image data obtained by extracting only the face of the subject is output. This is the face extraction image data P502 of FIG. 5B.
Then, the feature point extraction unit 402 performs processing such as polygon analysis on the face image of the subject included in the face extraction image data P502. Then, feature point data including the entire face of the subject, the contours of the eyebrows, eyes, nose, mouth, and the like and the feature points of the face indicating the pupil is generated. This is the feature point data P503 in FIG. 5C. This feature point data P503 is composed of a collection of feature points having coordinate information in a two-dimensional space.

When two sets of two-dimensional feature point data are acquired at different timings on the time axis, the face of the person to be photographed slightly moves, causing a shift in each feature point data. Based on this deviation, the direction of the face of the subject can be calculated. This is the face direction vector.
Moreover, the arrangement of the pupil with respect to the outline of the eye can calculate the approximate line-of-sight direction with respect to the face of the subject. This is the gaze direction vector.
The vector analysis unit 403 generates a face direction vector and a line-of-sight direction vector from the feature point data by the processing as described above.

As described above, in the wake-up monitoring device 104, the bed or the futon on which the elderly person 102 goes to sleep is arranged from the direction of the legs of the elderly person 102 toward the elderly person 102. For this reason, from the viewpoint of the wake-up monitoring device 104, the face of the elderly person 102 cannot be seen when the elderly person 102 goes to bed, and the face of the elderly person 102 can be seen when the elderly person 102 gets up. The state where the face of the elderly person 102 cannot be seen is a state where feature point data cannot be generated. Then, in this state, the vector analysis unit 403 cannot generate a face direction vector.
That is, the wake-up monitoring device 104 can detect whether or not the elderly person 102 is getting up by checking whether or not the vector analysis unit 403 can generate the face direction vector.

[Regarding eye-meditation determination unit 404]
FIG. 6A is a diagram in which the eye portion is extracted from the feature point data in a state in which the user opens the eye.
FIG. 6B is a diagram in which the eye portion is extracted from the feature point data in a state where the user closes the eye.
In the feature point data of FIG. 6A, some of the feature points indicate the outline of the eye and the pupil caused by the open eyelid. That is, the eye area A601 and A602 exist in the feature point data of FIG. 6A. For this reason, the vector analysis unit 403 can detect where the pupil points P603 and P604 exist in the eye regions A601 and A602, and generate a line-of-sight direction vector.

On the other hand, in the feature point data shown in FIG. 6B, the feature points represent the eyelid breaks L605 and L606 generated by the closed eyelid, and there is no eye region. For this reason, the vector analysis unit 403 cannot detect where the point indicating the pupil exists in the eye region. Therefore, the vector analysis unit 403 cannot generate a gaze direction vector.
Thus, the wake-up monitoring device 104 can detect whether the elderly person 102 has opened his eyes by checking whether or not the vector analysis unit 403 can generate the gaze direction vector.

In the present embodiment, the wake-up monitoring device 104 has been disclosed.
The wake-up monitoring device 104 is disposed at a position where the face of the elderly person 102 can be photographed while the elderly person 102 who is the subject is getting up. When the elderly person 102 gets up, the imaging device 308 and the near-infrared LED illuminating device 105 constituting the near-infrared camera take a picture. Then, when the wake-up monitoring device 104 detects the face direction vector and the gaze direction vector of the elderly person 102 from the image data stream output from the imaging device 308, and determines that the elderly person 102 is in an open state, An email and / or SNS message of a warning message is transmitted to the smartphone 110 of the staff member 109 who is in charge of monitoring.

  The wake-up monitoring device 104 in this embodiment does not use the face authentication collation technique disclosed in Patent Document 1 or the like. Therefore, the calculation processing is extremely light compared to the conventional method, and the need for prior registration of the face of the person being monitored in the system is unnecessary. Furthermore, the data transmitted by the wake-up monitoring device 104 is an e-mail and / or SNS message to which still image data is attached. Therefore, the amount of data that the wake-up monitoring device 104 sends to the LAN and / or the Internet 201 is extremely small as compared with a system that continuously transmits moving image data as in the prior art.

The embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and other modifications and application examples are provided without departing from the gist of the present invention described in the claims. including.
For example, the above-described embodiment is a detailed and specific description of the configuration of the apparatus and the system in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to one having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is also possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Each of the above-described configurations, functions, processing units, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Further, each of the above-described configurations, functions, and the like may be realized by software for interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files that realize each function must be held in a volatile or non-volatile storage such as a memory, hard disk, or SSD (Solid State Drive), or a recording medium such as an IC card or an optical disk. Can do.
In addition, the control lines and information lines are those that are considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 101 ... Elderly person facility, 102 ... Elderly person, 103 ... Room, 104 ... Wake-up monitoring device, 105 ... Near infrared LED lighting device, 106 ... Wireless LAN router, 107 ... Surveillance camera, 108 ... Nurse station, 109 ... Staff, 110 ... Smartphone, 111 ... Monitor, 201 ... Internet, 202 ... Mail server, 301 ... CPU, 302 ... ROM, 303 ... RAM, 304 ... Non-volatile storage, 305 ... Wireless LAN interface, 306 ... NIC, 307 ... Bus, 308 ... Imaging device 401 ... face detection processing unit 402 ... feature point extraction unit 403 ... vector analysis unit 404 ... eye-meditation determination unit 405 ... warning transmission unit 406 ... warning transmission information 407 ... interface selection unit

Claims (2)

An imaging device capable of photographing the face of the subject illuminated by near infrared rays;
A face detection processing unit that detects the presence of the face of the person to be photographed from an image data stream output from the imaging device, and outputs face extraction image data obtained by extracting the face of the person to be photographed;
A feature point extraction unit that outputs feature point data, which is a collection of feature points having coordinate information in a two-dimensional space, including the contour of the face of the subject, from the face extraction image data;
A vector analysis unit that generates, from the feature point data, a face direction vector indicating a direction of the face of the subject and a line-of-sight vector indicating the direction of the line of sight of the face of the subject;
When both the face direction vector and the line-of-sight direction vector are valid, an eye meditation determination unit that determines that the subject is in an open state;
The eye meditation determination unit includes a warning transmission unit that creates a warning message and transmits it to a predetermined device in response to the determination output result determined that the subject is in an open state. Wake-up monitoring device. The warning transmission unit attaches still image data of the subject to the warning message from the image data stream when creating the warning message.
The wake-up monitoring device according to claim 1.

Images