JP2018147278A - Solitary monitor system and mirror-like terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a solitary monitor system capable of satisfactorily monitoring a subject to be watched inexpensively without the necessity of a special skill such as conventional one of manipulating information equipment, and a mirror-like terminal to be employed in the system.SOLUTION: A mirror-like terminal images a solitary's face with a camera so as to acquire feature point data, an engagement value, and pulse rate data. The mirror-like terminal then transmits the data items to a portable wireless terminal of a family member, who resides separately, by way of a server. On receipt of the feature point data, engagement value, and pulse rate data, the server estimates the solitary's expression on the basis of the feature point data, and transmits a result of the expression estimation to the portable wireless terminal together with the feature point data, engagement value, and pulse rate data. The solitary is prompted to perform the work once a day.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a single person monitoring system particularly suitable for monitoring a single person living alone, and a mirror-type terminal used therefor.

  In recent years, in Japan, the aging population is declining and the generation of children and grandchildren is concentrated in large cities, and the form of families is increasing in which parents are left in rural areas. Along with this situation, the number of elderly living alone has increased. And the problems related to communication between the elderly living alone in rural areas and the separated families who are their child generations living in large cities are becoming more apparent. In particular, the lonely death of the elderly is a big problem.

In order to improve such a situation, Patent Document 1 discloses a monitoring system that can closely monitor a person to be monitored such as an elderly person or a disabled person.
Patent Document 2 discloses a technique for detecting a pulse using short-time Fourier transform (short-time Fourier transform, STFT) from image data of a user's face photographed by a camera.
Patent Document 3 discloses a technique for detecting a pulse using a discrete wavelet transform (DWT).

JP2015-108894A JP, 2015-116368, A Japanese Patent Laid-Open No. 10-216096

Patent applications and information related to a wide variety of watching systems have been disclosed so far, including the above-mentioned Patent Document 1. Most of them are modes in which an information device is used to monitor a person to watch over the Internet.
It is difficult to force the person to be watched, who are mostly vulnerable to information, to operate the information equipment.

  The present invention has been made in view of such a problem, and is a low-priced single person who can monitor a person to be watched as necessary and sufficiently without requiring a special skill for operating a conventional information device. It is an object of the present invention to provide a person monitoring system and a mirror type terminal used therefor.

In order to solve the above problems, a solitary person monitoring system of the present invention includes a mirror-type terminal that captures a face of a person to be monitored and outputs predetermined data, a server that receives predetermined data from the mirror-type terminal, It consists of a monitoring terminal that receives data output from a mirror type terminal transferred from a server.
The mirror-type terminal includes a mirror that can reflect the face of the monitoring target person, an imaging device that is incorporated in the mirror and captures the face of the monitoring target person and outputs image data, and the monitoring target person included in the image data A face detection processing unit for detecting the face of the subject, a feature point extraction unit for generating feature point data composed of the feature points of the face from the face extraction image data obtained by extracting the face of the person to be monitored output by the face detection processing unit, An input / output control unit connected to the network and transmitting the feature point data to the monitoring terminal via the server.
The server is connected to the mirror type terminal and the monitoring terminal through the network, and includes an input / output control unit that transfers feature point data transmitted from the mirror type terminal to the monitoring terminal.
The monitoring terminal includes an input / output control unit that receives the feature point data transferred from the server, and a display unit that displays the feature point data.

According to the present invention, a low-cost single person monitoring system capable of monitoring a person to be watched necessary and sufficiently without requiring special skills to operate a conventional information device, and a system for use in this system. A mirror-type terminal can be provided.
Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is the schematic which shows the state which installed and operated the single person monitoring system which concerns on embodiment of this invention in the single person's home. It is the perspective view and front view of the mirror type terminal which comprise the sensor terminal of the single person monitoring system which concerns on embodiment of this invention. It is a partial cross section exploded view of a mirror type terminal. It is a block diagram which shows the hardware constitutions of a mirror type terminal. It is a block diagram which shows the software function of a mirror type terminal. Schematic diagram showing an example of an image data stream output from the imaging device, schematic diagram showing an example of face extraction image data output by the face detection processing unit, and an example of feature point data output by the feature point extraction unit FIG. It is a figure which shows typically the area | region which a pulse detection area extraction part cuts out as partial image data from the image data of a single person's face. It is a block diagram which shows the hardware constitutions of a server. It is a block diagram which shows the software function of a server. It is the schematic explaining the classification of the emotion which the facial expression estimation part 905 implements. It is a block diagram which shows the hardware constitutions of a portable radio | wireless terminal. It is a block diagram which shows the software function of a portable radio | wireless terminal. It is the example of a display displayed on the display part of a portable radio | wireless terminal. It is a time chart which shows the procedure and content of the communication performed between a mirror type terminal and a portable radio | wireless terminal starting from the condition which detected that it was impossible to confirm the safety of a single person in a portable radio | wireless terminal. It is a block diagram which shows the software function of the server as a modification. It is a block diagram which shows the software function of the mirror type terminal as a modification.

The applicant has previously filed a patent application for a concentration processing system that captures a user viewing content with a camera and calculates the concentration based on the direction of the user's face and line of sight (Japanese Patent Application No. 2006-124611).
This concentration processing system captures a user's face with a camera, detects the orientation of the user's face and line of sight, and measures how much the orientation is directed to the display that displays the content. Calculate the degree of concentration on the content.
The single person monitoring system according to the embodiment of the present invention is constructed as an application for monitoring a single person based on this concentration processing system.
The concentration processing system described in the above-mentioned patent application also has a function of detecting a pulse from the user's face. The information on the pulse is useful information for estimating the general health status in addition to confirming the survival of the elderly living alone. The single person monitoring system according to the embodiment of the present invention also has a function of detecting a single person's pulse, like the concentration processing system of the previous application.

The inventors have so far developed an apparatus for measuring the degree of concentration. In the process of device development, the inventors have realized that the state where a person concentrates on a certain event includes not only active factors but also passive factors.
For example, the act of confronting a person and concentrating to solve the problem is an active factor. In other words, the act is attributed to the awareness that “we must concentrate on the event”. On the other hand, the act of attracting interest by seeing events such as things that are interesting or fun is a passive factor in a sense. In other words, the act is attributed to the feeling that “the event is unintentionally attracted”.

The inventors considered that it is not always appropriate to express the behavior caused by the conflicting consciousness and emotion in the word “concentration”. Therefore, the inventors decided to define the state of interest that a subject is interested in for an event, regardless of whether it is an active or passive factor, using the term “engagement”. . The inventors have defined the device that has been developed so far as a device that measures engagement, not a device that measures concentration.
The single person monitoring system described in this specification is a system that reduces the cost of hardware such as terminals and servers and the amount of data traveling on the network by diverting this engagement to single person monitoring. is there.

[Installation operation example]
FIG. 1 is a schematic diagram showing a state in which a solitary person monitoring system 101 according to an embodiment of the present invention is installed and operated in a residence 103 of a solitary person 102.
In the home 103 of the solitary person 102 who is the monitoring subject, the solitary person 102 is looking at the mirror type terminal 104 constituting the sensor terminal. A separate family 105 of a solitary person 102 living in an urban area who is a supervising person to be monitored asks the solitary person 102 to “look at this mirror once a day”. When the solitary person 102 looks at the mirror type terminal 104 and confirms his / her face, the imaging device embedded in the mirror type terminal 104 recognizes the existence of the face of the solitary person 102. Furthermore, the pulse is detected from the image information of the face of the solitary person 102. Then, information indicating that the solitary person 102 has come in front of the mirror-type terminal 104 and pulse information are transmitted to the server 107 via the Internet 106.
When the server 107 receives information from the mirror-type terminal 104, the server 107 registers the information in the included database. Then, the information sent from the mirror-type terminal 104 is transferred to the portable wireless terminal 108 which is a monitoring terminal owned by the separated family 105 of the single person 102. The separated family 105 of the solitary person 102 recognizes that the portable wireless terminal 108 has received the information, and grasps the safety of the solitary person 102 and the general health condition.

FIG. 2A is a perspective view of the mirror type terminal 104 constituting the sensor terminal of the single person monitoring system 101 according to the embodiment of the present invention, and FIG. 2B is a front view of the mirror type terminal 104.
As shown in FIG. 2A, the mirror-type terminal 104 constitutes a table mirror used by being placed on a desk or table. The mirror-type terminal 104 includes a mirror surface portion 201 having an inclination angle of about 60 ° and a support body 202 that supports the mirror surface portion 201 in consideration of use as a mirror.
As shown in FIGS. 2A and 2B, a push button switch 203 in which an LED is embedded is provided on the upper right side of the mirror surface portion 201. The LED embedded in the push button switch 203 blinks or lights when alarm information transmitted from the portable wireless terminal 108 or the server 107 owned by the separate family 105 is received.
Further, the mirror of the mirror surface portion 201 is formed of a half mirror, and an information processing portion 304 (see FIG. 3) having an imaging device is embedded in a dotted line portion P204 shown in FIGS. 2A and 2B. The imaging device itself is difficult to see with the naked eye due to the presence of the half mirror.

FIG. 3 is a partially exploded sectional view of the mirror-type terminal 104.
A half mirror film 302 is attached to the back surface of the transparent top plate 301 and functions as a mirror that reflects the face of the lonely person 102. The top plate 301 is a transparent resin such as acrylic, polycarbonate, polyethylene terephthalate, or hard vinyl chloride, or a transparent plate such as tempered glass, and may have a corresponding strength. The half mirror film 302 is a film for forming a half mirror by being attached to a transparent plate such as acrylic or glass, and is made of a material such as a dielectric multilayer film or a metal thin film. This film is also known as a magic mirror.
The top plate 301 to which the half mirror film 302 is attached is fitted into a frame 303 made of synthetic resin or the like. A hole 303a is formed in the frame 303, and an information processing unit 304 having an imaging device is inserted into the hole 303a. An optical system of the imaging device is provided in a portion 304a of the information processing unit 304 that is inserted into the hole 303a.

[Mirror type terminal 104: hardware configuration]
FIG. 4 is a block diagram illustrating a hardware configuration of the information processing unit 304 of the mirror type terminal 104.
The information processing unit 304 of the mirror-type terminal 104 configured by a single board computer includes a CPU 401, a ROM 402, a RAM 403, a non-volatile storage 404, and a real-time clock (hereinafter referred to as “RTC”) that outputs current date and time information. 405, a wireless LAN interface 406, and a wireless communication unit 407. Further, a push button switch 203, an LED 410, and a speaker 411 are connected to the bus 408 via a serial interface 409.
The wireless LAN interface 406 is a network interface for connecting to the Internet 106 via a wireless LAN router (not shown).
The wireless communication unit 407 includes a transmission circuit and a reception circuit for wireless communication, and is a network interface for connecting to the Internet 106 through an access point (not shown). For example, the wireless communication unit 407 is connected to the access point according to a communication standard such as LTE (Long Term Evolution). Connected. An imaging device 412 that captures the face of the solitary person 102 is also connected to the bus 408.

The non-volatile storage 404 stores a network OS including a TCP / IP protocol stack for connecting the information processing unit 304 to the network, and a program for operating as the mirror terminal 104.
A speaker 411 connected to the serial interface 409 generates an alarm sound without using a D / A converter, based on a PWM signal supplied via a coupling capacitor (not shown).

  For example, “Raspberry Pi” developed by the Raspberry Pi Foundation (http://www.raspberrypi.org/) in the UK can be used as the single board computer constituting the information processing unit 304. The arithmetic processing capability of the information processing unit 304 is such that a network OS such as Linux (registered trademark) is used at a practical speed in order to realize a function of connecting to the Internet 106 and a function of executing processing of image data described later. It is only necessary to be able to operate.

  The imaging device 412 takes into consideration the angle of view and the like so that the single person 102 who is the subject of monitoring looks into the mirror and can capture the face of the single person 102 sufficiently and sufficiently. As an example, an imaging device 412 having an angle of view of 28 ° and an XGA (1024 × 768 pixels) or more is used. The optical system of the imaging device 412 preferably has a pan focus specification in order to eliminate the need for focus adjustment. In addition, since the amount of light is greatly reduced by the half mirror film 302, a highly sensitive imaging device 412 is desirable.

[Mirror type terminal 104: Software function]
FIG. 5 is a block diagram showing software functions of the information processing unit 304 of the mirror-type terminal 104 according to the embodiment of the present invention.
The image data stream output from the imaging device 412 is supplied to the face detection processing unit 501.
The face detection processing unit 501 regards the image data stream output from the imaging device 412 as a still image continuous on the time axis, and performs, for example, the Viola-Jones method on each image data of the still image continuous on the time axis. The presence of the face of the solitary person 102 is detected using a known algorithm such as. And the face extraction image data which extracted only the face of the single person 102 is output.

The face extraction image data output from the face detection processing unit 501 is supplied to the feature point extraction unit 502.
The feature point extraction unit 502 performs processing such as polygon analysis on the face image of the single person 102 included in the face extraction image data. Then, feature point data including the entire face of the solitary person 102, the outlines of the eyebrows, eyes, nose, mouth, and the like and the feature points of the face indicating the pupil is generated. Details of the feature point data will be described later with reference to FIG.

The feature point data output from the feature point extraction unit 502 is supplied to the vector analysis unit 503.
The vector analysis unit 503 obtains a vector indicating the orientation of the face of the solitary person 102 (hereinafter referred to as “face direction vector”) from the feature point data based on the two consecutive face extraction image data, and the line of sight of the face of the solitary person 102. A vector indicating the direction of the image (hereinafter referred to as a “gaze direction vector”) is generated.

The face direction vector and the line-of-sight direction vector are supplied to the engagement calculation unit 504.
The engagement calculation unit 504 adds the face direction vector and the line-of-sight direction vector, and indicates a gaze direction vector indicating where the single person 102 is gazing in the three-dimensional space including the display that displays the content and the imaging device 412. Is calculated. And it is determined whether the gaze direction of the single person 102 is suitable for the display. This determination result is a binary value indicating whether the gaze direction of the solitary person 102 is facing the display (logical “true”) or not (logical “false”).
The engagement calculation unit 504 calculates the gaze direction determination result every 100 msec, for example. Then, for example, the moving average of the gaze direction determination results for the latest one second is calculated. This calculation makes it possible to acquire a binary gaze direction determination result as a pseudo-analog value. The moving average value of the gaze direction determination result is the engagement value.

The face extraction image data output from the face detection processing unit 501 and the feature point data output from the feature point extraction unit 502 are also supplied to the pulse detection region extraction unit 505.
The pulse detection area extraction unit 505 is an image corresponding to a part of the face of the solitary person 102 based on the face extraction image data output from the face detection processing unit 501 and the feature point data output from the feature point extraction unit 502. Data segmentation is executed, and the obtained partial image data is output to the pulse calculation unit 506. Here, the extraction of image data corresponding to a part of the face of the solitary person 102 performed by the pulse detection area extracting unit 505 is right under the eyes of the face of the solitary person 102, which is suitable as an area for detecting the pulse. A region corresponding to the cheekbone is preferred.

  As a region for detecting a pulse in the pulse detection region extraction unit 505, a region slightly above the lips and the eyebrows and the vicinity of the cheekbone may be considered. A small area near the cheekbone is used. Various methods can be considered for determining the pulse detection region. For example, it may be slightly above the lips or between the eyebrows. Furthermore, it is possible to analyze a plurality of candidate areas such as immediately above the lips and between the eyebrows and the vicinity of the cheekbones. If the lips are hidden in the eyelid, the next candidate (for example, immediately above the eyebrows) and the next candidate are also hidden. If so, the candidates may be narrowed down in order, such as the next candidate (near the cheekbone), and an appropriate cutout area determined.

The pulse calculation unit 506 extracts (filters) a green component from the partial image data generated by the pulse detection region extraction unit 505, and obtains an average value of luminance for each pixel. Then, the pulse of the solitary person 102 is calculated using the short-time Fourier transform described in Patent Document 2 or the like, or the discrete wavelet transform described in Patent Document 3 or the like, for example, with respect to the fluctuation of the average value. . Note that the pulse calculation unit 506 of the present embodiment obtains an average value of luminance for each pixel, but a mode value or a median value may be adopted in addition to the average value.
It is known that hemoglobin contained in blood has a characteristic of absorbing green light. A known pulse oximeter utilizes the characteristics of this hemoglobin, irradiates the skin with green light, detects reflected light, and detects a pulse based on the intensity change. The pulse calculation unit 506 is the same in that the characteristics of the hemoglobin are used. However, it differs from a pulse oximeter in that the data that becomes the basis for detection is image data.

The feature point data output from the feature point extraction unit 502, the engagement value output from the engagement calculation unit 504, and the pulse data output from the pulse calculation unit 506 are supplied to the input / output control unit 507.
The input / output control unit 507 adds the current date and time information output by the RTC 405 and the information stored in the server access information 508 to the feature point data, the engagement value, and the pulse data, and generates a transmission packet.
Server access information 508 stored in the non-volatile storage includes
-Monitoring target ID information for uniquely identifying the solitary person 102 who is the information transmission source,
・ Authentication information at the information source, such as passwords necessary for ensuring communication security,
An FQDN (Full Qualified Domain Name) or IP address of the server 107 for accessing the server 107
Monitor information that uniquely identifies the separated family 105 that is the information transmission destination is included.

That is, the input / output control unit 507 forms the message of feature point data, engagement value, pulse data, current date / time information, monitoring target ID information, and supervisor ID information, and then accesses the server 107 using the FQDN of the server 107. To do. Then, after obtaining authentication from the server 107 using the authentication information, the transmission packet generated from the message is transmitted to the server 107.
The transmission packet is transmitted through the interface selection unit 509 to the server 107 connected to the Internet 106 using the wireless LAN interface 406 or the HTTP (Hyper Text Transfer Protocol) of the wireless communication unit 407. That is, the input / output control unit 507 has a web client function.

The substance of the interface selection unit 509 is a TCP / IP protocol stack and a DHCP (Dynamic Host Configuration Protocol) client provided in the OS. That is, the interface selection unit 509 selects a network interface connected to the IP reachable network, and transmits the transmission packet to the server 107.
In the mirror type terminal 104 according to the embodiment of the present invention, the simplest and easy-to-handle HTTP is exemplified as the protocol used for communication in the network, but the protocol for transmitting the data stream is not limited to this. . For example, the well-known SMTP (Simple Mail Transfer Protocol), that is, Internet 106 electronic mail may be used.

  The input / output control unit 507 can receive data transmitted from the server 107 immediately by holding the connection (TCP connection) with the server 107 without disconnecting. Hereinafter, this communication method is abbreviated as “push-type communication” (HTTP server push). When an alarm message is received from the server 107, the alarm detection unit 510 provided in the input / output control unit 507 detects the reception of the alarm. In response to the alarm detection unit 510 detecting the reception of the alarm, the input / output control unit 507 drives the LED 410 to emit light and drives the speaker 411 to sound. When it is detected that the single person 102 has operated the push button switch 203, the input / output control unit 507 stops the light emission driving of the LED 410.

[About transmission data]
In addition to the feature point data, the engagement value, and the pulse data measured every 100 msec, the transmission packet includes current date and time information, monitoring target ID information, and supervisor ID information output by the RTC 405. Such information is recorded in the database of the server 107. In order to make the information recorded in the database as accurate as possible, it is preferable to keep the RTC 405 accurate. For this purpose, it is preferable to install and execute a program having a date and time information calibration function such as an NTP (Network Time Protocol) client in the mirror type terminal 104.

  Further, since the server 107 receives information from the plurality of mirror-type terminals 104, information for uniquely identifying each mirror-type terminal 104 is necessary. For this reason, monitoring target ID information for uniquely identifying the mirror terminal 104 and / or the single person 102 is provided. Further, the server 107 needs monitor ID information in order to transfer the information received from the mirror type terminal 104 to the correct monitor.

[About feature point data]
The operations of the face detection processing unit 501, the feature point extraction unit 502, and the vector analysis unit 503 will be described.
FIG. 6A is a schematic diagram illustrating an example of an image data stream output from the imaging device 412. FIG. 6B is a schematic diagram illustrating an example of face extraction image data output by the face detection processing unit 501. FIG. 6C is a schematic diagram illustrating an example of feature point data output by the feature point extraction unit 502.
First, an image data stream including the subject 601 is output from the imaging device 412 in real time. This is the image data P602 in FIG. 6A.

Next, the face detection processing unit 501 detects the presence of the face of the subject 601 from the image data P601 output from the imaging device 412 using a known algorithm such as the Viola-Jones method. Then, face extraction image data obtained by extracting only the face of the subject 601 is output. This is the face extraction image data P603 in FIG. 6B.
Then, the feature point extraction unit 502 performs processing such as polygon analysis on the face image of the subject 601 included in the face extraction image data P603. Then, feature point data including the entire face of the subject 601, the contours of the eyebrows, eyes, nose, mouth, and the like and the feature points of the face indicating the pupil is generated. This is the feature point data P604 in FIG. 6C. This feature point data P604 is composed of a collection of feature points having coordinate information in a two-dimensional space.

When two sets of two-dimensional feature point data are acquired at different timings on the time axis, the face of the person to be photographed 601 slightly moves, causing a shift in each feature point data. Based on this deviation, the face direction of the subject 601 can be calculated. This is the face direction vector.
In addition, the arrangement of the pupil with respect to the outline of the eye can calculate the approximate line-of-sight direction with respect to the face of the subject 601. This is the gaze direction vector.
The vector analysis unit 503 generates a face direction vector and a line-of-sight direction vector from the feature point data by the processing as described above.

[About pulse detection area]
FIG. 7 is a diagram schematically showing a region that the pulse detection region extraction unit 505 cuts out as partial image data from the image data of the face of the solitary person 102.
As described in Patent Document 2, in order to correctly detect the pulse from the color of the skin of the face, the color of the skin such as eyes, nostrils, lips, hair and wrinkles in the face image data is included. It is necessary to eliminate as much as possible the elements that are unrelated to. In particular, the eyes move rapidly, and when the eyelids are closed or opened, a rapid luminance change occurs in a short time such as the presence or absence of pupils in the image data. For this reason, the eye information has an adverse effect when calculating the average value of luminance. In addition, although there are individual differences, the presence of hair and wrinkles is a factor that greatly hinders the detection of skin color.
Considering the above, as shown in FIG. 7, the regions 701a and 701b under the eyes are not easily affected by the presence of eyes, hair, and wrinkles. It is considered suitable as an example.
In the solitary person monitoring system 101 according to the embodiment of the present invention, since the face of the solitary person 102 is vectorized and has the function of recognizing the face of the solitary person 102, the pulse detection area extracting unit 505 has the face feature points. It is possible to calculate the coordinate information of the area under the eyes from the eye.

[Server 107: Hardware configuration]
FIG. 8 is a block diagram illustrating a hardware configuration of the server 107.
In the server 107, a CPU 801, a ROM 802, a RAM 803, a nonvolatile storage 804, an RTC 805 and a NIC 806 are connected to a bus 807.
The non-volatile storage 804 stores a network OS including a TCP / IP protocol stack for connecting the server 107 to the Internet 106 and a program for causing the server 107 to function as a computer.

[Server 107: Software function]
FIG. 9 is a block diagram illustrating software functions of the server 107.
The server 107 has a function as a web server that transmits and receives HTTP messages to and from the mirror terminal 104 and the portable wireless terminal 108, a function as a database server that registers data received from the mirror terminal 104 in a table, It has a function as an application server that estimates the expression of the solitary person 102 from the feature point data received from the terminal 104.
The entity of the input / output control unit 901 is a web server program, which receives a request from the mirror type terminal 104 or the portable wireless terminal 108 that is an HTTP client, and returns response data corresponding to the request.

When the input / output control unit 901 receives a connection request from the mirror terminal 104, the input / output control unit 901 performs user authentication through the authentication processing unit 902. At that time, the authentication processing unit 902 refers to the user master 903.
Information including the feature point data, the engagement value, and the pulse data transmitted from the mirror-type terminal 104 that has been successfully authenticated is sent to the log table 904 provided in the nonvolatile storage 804 via the input / output control unit 901. To be recorded.
Next, the input / output control unit 901 passes the feature point data included in the data received from the mirror terminal 104 to the facial expression estimation unit 905. The facial expression estimation unit 905 estimates the facial expression of the solitary person 102 from the feature point data using a learning algorithm (deep learning) suitable for image recognition represented by a known convolutional neural network or the like. The input / output control unit 901 registers the facial expression estimation result estimated by the facial expression estimation unit 905 in the same record of the table as the data received from the mirror terminal 104. The input / output control unit 901 transmits information including the feature point data, the engagement value, the pulse data, and the facial expression estimation result to the portable wireless terminal 108 owned by the separate family 105.

The first purpose of the solitary person monitoring system 101 according to the embodiment of the present invention is that when the mirror-type terminal 104 detects the face of the solitary person 102, the pulse is measured from the face image of the solitary person 102, and appropriate information is obtained. It is to transmit to the portable wireless terminal 108 owned by the separate family 105.
Incidentally, both the mirror type terminal 104 and the portable wireless terminal 108 correspond to clients on the Internet 106. In particular, client devices installed in the home 103 are often present inside a firewall, which is a function of a wireless LAN router or the like, from the viewpoint of the network. Direct P2P connection between client devices concealed by a firewall and assigned a private IP address is extremely difficult.

Therefore, in order for clients to communicate with each other, it is necessary to have a server 107 having a global IP address and preferably a defined FQDN.
That is, the server 107 plays a role of mediating communication between the mirror terminal 104 and the portable wireless terminal 108. Then, in order to quickly realize communication between the client devices, the server 107 executes push-type communication with respect to each of the mirror type terminal 104 and the portable wireless terminal 108. By executing push communication to both client devices, highly immediate communication is realized for the two client devices.
It should be noted that this is not the case when IPv6 spreads over the Internet 106 in Japan and around the world, or when P2P communication between client devices can be easily realized by some means such as an overlay network.

[Emotion estimation]
FIG. 10 is a schematic diagram for explaining emotion classification performed by the facial expression estimation unit 905.
According to Paul Ekman, human beings have universal feelings no matter what language or culture they belong to. The classification of emotions by Ekman is also called “Ekman's basic six emotions”. Human expression changes in six emotions: surprise (F1002), fear (F1003), disgust (F1004), anger (F1005), joy (F1006), and sadness (F1007), compared to normal expressionless (F1001). To do. Changes in facial expressions appear as changes in facial feature points. The facial expression estimation unit 905 detects the relative variation of the facial feature points on the time axis, and estimates which emotion the facial expression of the solitary person 102 belongs to according to Ekman's six basic emotions.

A known learning algorithm used for image recognition performs arithmetic processing after performing preprocessing such as extracting image data as it is or only a predetermined color component. In many cases, the work of greatly reducing the number of pixels is not included.
The facial expression estimation unit 905 of the server 107 targets facial feature point data, not image data. This feature point data is only about 10 to 30 pieces of coordinate information, and the amount of data is overwhelmingly small compared to image data which is a large amount of data including luminance information for each pixel. However, the feature point data alone has an information amount enough to sufficiently estimate the change in facial expression shown in FIG. That is, the change in the human facial expression can be sufficiently estimated only by the feature point data. For this reason, if this feature point data is input to image recognition processing using an existing learning algorithm, the amount of calculation is significantly reduced compared to the conventional case, and an estimation result can be obtained quickly. Also, the load on the server 107 is very light.

[Portable wireless terminal 108: hardware configuration]
FIG. 11 is a block diagram illustrating a hardware configuration of the portable wireless terminal 108.
A portable wireless terminal 108 serving as a monitoring terminal includes a CPU 1101, a ROM 1102, a RAM 1103, a display unit 1104, an operation unit 1105 that is an electrostatic position detection device, a nonvolatile storage 1106, an RTC 1107, connected to a bus 1110. A wireless LAN I / F 1108 and a wireless communication unit 1109 are provided. The display unit 1104 and the operation unit 1105 constitute a known touch panel display.
The nonvolatile storage 1106 stores a network OS including a TCP / IP protocol stack for connecting the server 107 to the Internet 106 and a program for operating as the portable wireless terminal 108.
In the single person monitoring system 101 according to the embodiment of the present invention, the monitoring terminal used by the separate family 105 is the portable wireless terminal 108, but the monitoring terminal is not limited to the portable wireless terminal 108. A general personal computer may be used. In FIG. 13 described later, a display screen of the portable wireless terminal 108 will be described. A tap operation on the touch panel display of the portable wireless terminal 108 corresponds to a click operation with a mouse in a personal computer.

[Portable wireless terminal 108: Software function]
FIG. 12 is a block diagram illustrating software functions of the portable wireless terminal 108.
Similar to the mirror terminal 104, the input / output control unit 1201 of the portable wireless terminal 108 also has a web client function. The input / output control unit 1201 authenticates the server 107, which is a web server, using authentication information stored in the server access information 1202. When the authentication ends normally, the input / output control unit 1201 establishes a connection with the server 107 and performs push-type communication. By executing the push type communication, the information transmitted from the mirror type terminal 104 can be quickly received.

Similarly to the mirror type terminal 104, the input / output control unit 1201 of the portable wireless terminal 108 accesses the Internet 106 through the interface selection unit 1203 using either the wireless LAN I / F 1108 or the wireless communication unit 1109. , Communication with the server 107 is realized.
The engagement value, pulse data, facial expression estimation result, and date / time information of the single person 102 received from the mirror type terminal 104 through the server 107 are recorded in a log table 1204 provided in the nonvolatile storage 1106.

The server access information 1202 stored in the nonvolatile storage 1106 includes
Monitor ID information that uniquely identifies the separated family 105 that is the information transmission source,
・ Authentication information at the information source, such as passwords necessary for ensuring communication security,
An FQDN (Full Qualified Domain Name) or IP address of the server 107 for accessing the server 107
Monitoring target ID information that uniquely identifies the solitary person 102 who is the information transmission destination is included.
The input / output control unit 1201 uses authentication information in order to establish a connection with the server 107 and perform push communication.

  When the alarm transmission processing unit 1205 recognizes that the information received from the mirror type terminal 104 through the server 107 is abnormal, or when determining that the information cannot be received from the mirror type terminal 104 at a preset date and time, An alarm is generated and transmitted to the mirror type terminal 104 through the input / output control unit 1201. Therefore, the alarm transmission processing unit 1205 monitors the current date / time information output by the RTC 1107.

[Portable wireless terminal 108: display example]
13A, 13B, and 13C are display examples displayed on the display unit 1104 of the portable wireless terminal 108. FIG.
FIG. 13A is a display example that appears on the touch panel display of the portable wireless terminal 108 of the separate family 105 when the single person 102 comes in front of the mirror type terminal 104 and looks into the mirror type terminal 104.
The display field displayed at the top of the screen is the name display field P1301 of the solitary person 102. In FIG. 13A, FIG. 13B, and FIG. 13C, “Grandma” is displayed in the name display column P1301, but the name of the solitary person 102 is actually displayed.

The display field at the upper left is a face outline display field P1302 for displaying the face outline of the solitary person 102 based on the feature point data of the face of the solitary person 102 received from the mirror type terminal 104 through the server 107. Since the feature point data has a significantly smaller data amount than the image data, the feature point data is hardly affected by the congestion state of the Internet 106 and can be displayed in real time.
The display column on the upper right is a facial expression estimation result display column P1303 that displays a simple illustration of facial expression estimation results and facial expressions estimated from the feature point data of the face of the solitary person 102 received from the server 107.

The lower left display column is a pulse display column P1304 that displays the pulse data and pulse waveform of the solitary person 102 received from the mirror terminal 104 through the server 107.
The display field in the lower right is an engagement value display field P1305 that displays the engagement value of the solitary person 102 and the line graph of the engagement value received from the mirror terminal 104 through the server 107.
If the single person 102 is in a normal state, a display screen as shown in FIG. 13A can be confirmed. These data amounts are extremely small compared to image data. Therefore, the communication cost required for the mirror type terminal 104 and the portable wireless terminal 108 can be reduced.

  FIG. 13B shows the touch panel of the portable wireless terminal 108 of the separated family 105 when the single person 102 does not come in front of the mirror type terminal 104 during the time or period (for example, two days) promised in advance from the separated family 105. It is an example of a display of the alarm screen which appears on a display. FIG. 13C is a display example of a safety confirmation screen that appears on the touch panel display of the portable wireless terminal 108 of the separated family 105 when the safety of the single person 102 can be confirmed. 13B and 13C will be described with reference to the time chart shown in FIG.

FIG. 14 shows the procedure and contents of communication performed between the mirror terminal 104 and the portable wireless terminal 108, starting from the situation where it is detected that the safety of the single person 102 cannot be confirmed in the portable wireless terminal 108. It is a time chart shown.
The portable wireless terminal 108 of the separate family 105 sets a preset time (for example, from 5:00 to 8:00 in the morning) or a period (for example, two days) as the monitoring time zone for the single person 102. During this monitoring period, when the single person 102 comes in front of the mirror terminal 104 and looks into the mirror terminal 104, the contents shown in FIG. 13A are displayed on the touch panel display of the portable wireless terminal 108. Is done.

  One day, the portable wireless terminal 108 starts monitoring at the start time of the monitoring time zone, and waits for reception of predetermined data from the mirror terminal 104 (S1401). If the predetermined data cannot be received even when the monitoring time period ends (S1402), the input / output control unit 1201 of the portable wireless terminal 108 displays the alarm screen shown in FIG. 13B (S1403). It waits for the operation of the separate family 105 who is the user of the type wireless terminal 108. When the separated family 105 taps the Call button (S1404), the input / output control unit 1201 of the portable wireless terminal 108 transmits an alarm to the mirror terminal 104 (S1405).

Here, the display content of the alarm screen shown in FIG. 13B will be described.
In FIG. 13B, a warning message column P1306 displayed on the touch panel display displays text indicating how long the single person 102 has not looked at the mirror-type terminal 104, a CALL button B1307, and a LIVE button B1308. Has been.
When the CALL button B 1307 is tapped, the portable wireless terminal 108 transmits a warning message toward the mirror terminal 104. When the input / output control unit 507 of the mirror-type terminal 104 receives a warning message from the portable wireless terminal 108 through the server 107, the LED 410 incorporated in the pushbutton switch 203 flashes and emits an alarm sound from the speaker 411.
When the LIVE button B 1308 is tapped, the portable wireless terminal 108 transmits a LIVE operation command to the mirror terminal 104. When the input / output control unit 507 of the mirror type terminal 104 receives the LIVE operation command from the portable wireless terminal 108 through the server 107, the input / output control unit 507 activates the imaging device 412 and generates moving image data with reduced resolution from the captured image data stream. And transmitted to the portable wireless terminal 108 through the server 107. Then, a LIVE moving image display area P1309 is formed on the lower side of FIG. 13B, and the LIVE moving image, that is, the state of the current living alone 103 in the home 103 is displayed.

Returning to FIG. 14, the description of the time chart will be continued.
When the input / output control unit 507 of the mirror type terminal 104 receives the alarm information transmitted from the portable wireless terminal 108 through the server 107 (S1406), the input / output control unit 507 controls blinking of the LED 410 and rings the speaker 411 (S1407). And it waits for the operation of the single person 102 who is a user. When the solitary person 102 operates the push button switch 203 (S1408), the input / output control unit 507 of the mirror-type terminal 104 stops ringing of the speaker 411 (S1409). Then, a reply is transmitted to the portable wireless terminal 108 (S1410).

When the input / output control unit 1201 of the portable wireless terminal 108 receives the reply information transmitted from the mirror-type terminal 104 through the server 107 (S1411), the safety confirmation screen shown in FIG. 13C is displayed (S1412), and the separated family Wait for operation 105. When the separate family 105 taps the OK button (S1413), the input / output control unit 1201 of the portable wireless terminal 108 transmits a confirmation end message to the mirror terminal 104 (S1414).
When the input / output control unit 507 of the mirror type terminal 104 receives the confirmation end message transmitted from the portable wireless terminal 108 through the server 107 (S1415), the input / output control unit 507 stops the blinking of the LED 410 (S1416).

Here, the display content of the safety confirmation screen shown in FIG. 13C will be described.
In FIG. 13C, in the message column P1310 displayed on the touch panel display, a text indicating that the safety of the single person 102 has been confirmed and an OK button B1311 are displayed.
When the OK button B1311 is tapped, the portable wireless terminal 108 transmits a confirmation end message to the mirror terminal 104. When the input / output control unit 507 of the mirror-type terminal 104 receives the confirmation end message from the portable wireless terminal 108 through the server 107, the input / output control unit 507 stops the flashing light emission of the LED 410 incorporated in the push button switch 203.

The embodiment described above can be applied as follows.
(1) Although the name of the solitary person monitoring system 101 according to the embodiment of the present invention can be regarded as “monitoring the solitary person 102”, the monitoring target of this system is not necessarily limited to the solitary person 102 alone. The monitoring target may be an elderly couple or a small group of care recipients.
(2) In the time chart of FIG. 14, the portable wireless terminal 108 performs monitoring of the single person 102 during the monitoring time period, but the server 107 may execute monitoring of the single person 102 during the monitoring time period.

(3) The mirror-type terminal 104 has acquired the feature point data, the engagement value, and the pulse data from the face image of the solitary person 102. Alternatively, only the face detection result of the face detection processing unit may be transmitted. Or the form which only transmits a feature point extraction part may be sufficient. The safety confirmation process described with reference to FIGS. 13B, 13C, and 14 can be executed even with the configuration of only the face detection result or the feature point data.
(4) Further, the mirror-type terminal 104 has acquired the feature point data, the engagement value, and the pulse data from the face image of the solitary person 102, but if the objective of monitoring the health condition of the solitary person 102 is satisfied, The engagement value may be omitted.

(5) Furthermore, the mirror-type terminal 104 collects biological information of the lonely person 102 from other biological information measuring devices such as a blood pressure monitor and an activity meter using short distance wireless communication such as BlueTooth (registered trademark). The feature point data, the engagement value, and the pulse data may be transmitted to the server 107 together.
(6) A microphone may be connected to the mirror-type terminal 104 and provided with an IP telephone function so that the solitary person 102 can make a call with the separated family 105 at the time of safety confirmation.
(7) In the above-described embodiment, the facial expression estimation unit 905 is provided in the server 107. Facial expression estimation processing, which is also an application of image matching processing, is a fairly heavy category for computer processing. However, if the computing capability of the information processing unit 304 of the mirror-type terminal 104 is sufficiently high and the computer resources necessary for the facial expression estimation process can be reduced, the facial expression estimation unit 905 may be provided in the mirror-type terminal 104.

(8) In addition, facial expression estimation processing using a learning algorithm requires computer resources mainly for feature amount calculation processing. Therefore, a feature value is created on the server side, and the mirror type terminal downloads the feature value created by the server from the server, and the mirror type terminal performs only the facial expression estimation process. Can realize facial expression estimation processing with high speed and accuracy.
FIG. 15 is a block diagram illustrating software functions of the server 1501 as a modification.
The server 1501 shown in FIG. 15 is different from the server 107 shown in FIG. 9 in that a facial expression learning unit 1502 and a feature quantity 1503 are provided instead of the facial expression estimation unit 905.
The input / output control unit 901 delivers the feature point data included in the data received from the mirror-type terminal 1601 (see FIG. 16) to the facial expression learning unit 1502. The facial expression learning unit 1502 creates a feature quantity 1503 for estimating the expression of the solitary person 102 from the feature point data by using a learning algorithm (deep learning) suitable for image recognition represented by a known convolutional neural network or the like. And update.
Each time the feature amount 1503 is updated, the mirror terminal 1601 is downloaded from the server 1501.

FIG. 16 is a block diagram illustrating software functions of a mirror-type terminal 1601 as a modified example.
The mirror type terminal 1601 shown in FIG. 16 is different from the mirror type terminal 104 shown in FIG. 5 in that a feature amount 1503 and a facial expression estimation unit 1602 are provided.
The facial expression estimation unit 1602 reads the feature point data output from the feature point extraction unit 502, refers to the feature amount 1503 downloaded from the server 1501, and estimates the facial expression of the solitary person 102. The input / output control unit 507 forms the feature point data, the engagement value, the pulse data, the facial expression estimation result, the current date / time information, the monitoring target ID information, and the supervisor ID information as a message, and then uses the FQDN of the server 1501 to generate the server 1501. To access. Then, after obtaining authentication from the server 1501 using the authentication information, the transmission packet generated from the message is transmitted to the server 1501.

In this embodiment, the single person monitoring system 101 was disclosed.
The mirror-type terminal 104 acquires feature point data, an engagement value, and pulse data when the imaging device 412 captures the face of the solitary person 102. Then, these data are transmitted to the portable wireless terminal 108 of the separated family 105 through the server 107. When the server 107 receives the feature point data, the engagement value, and the pulse data from the mirror type terminal 104, the server 107 estimates the expression of the solitary person 102 from the feature point data, and the facial expression estimation result together with the feature point data, the engagement value, and the pulse data. Transmit to the wireless terminal 108. This operation is performed once a day by a single person 102 at any time during a preset monitoring time period or without specifying a time. When there is no face photographing operation of the solitary person 102 during the monitoring time period, the portable wireless terminal 108 or the server 107 recognizes that it is in an abnormal state, and transmits an alarm to the mirror terminal 104. If the solitary person 102 is in a normal state, the solitary person 102 looks into the mirror-type terminal 104 by the blinking LED and the alarm sound of the mirror-type terminal 104 that has received the alarm. By doing so, the safety of the solitary person 102 can be confirmed.

The arithmetic processing of the mirror type terminal 104 in this embodiment is extremely light compared to the conventional one. Further, the data transmitted by the solitary person monitoring system 101 is feature point data, engagement value, and pulse data with extremely small data capacity. Therefore, the amount of data that the solitary person monitoring system 101 sends to the Internet 106 is extremely small.
Unlike conventional technology that transmits image data itself, image data is not always transmitted, but instead feature point data with a very small amount of data is transmitted, so that the privacy of a single person 102 on the Internet 106 is highly concealed. can do. Even if tapping (tapping: eavesdropping on the network) is performed, it is impossible to specify the solitary person 102 only by the feature point data.
Furthermore, the fact that the amount of data transmitted and received in the mirror type terminal 104 and the portable wireless terminal 108 is small means that the cost required for communication is low.

The embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and other modifications and application examples are provided without departing from the gist of the present invention described in the claims. including.
For example, the above-described embodiment is a detailed and specific description of the configuration of the apparatus and the system in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to one having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is also possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Each of the above-described configurations, functions, processing units, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Further, each of the above-described configurations, functions, and the like may be realized by software for interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files that realize each function must be held in a volatile or non-volatile storage such as a memory, hard disk, or SSD (Solid State Drive), or a recording medium such as an IC card or an optical disk. Can do.
In addition, the control lines and information lines are those that are considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 101 ... Single person monitoring system, 102 ... Single person, 103 ... Home, 104 ... Mirror type terminal, 105 ... Separate family, 106 ... Internet, 107 ... Server, 108 ... Portable wireless terminal, 201 ... Mirror surface part, 202 ... Support Body, 203 ... push button switch, 301 ... top plate, 302 ... half mirror film, 303 ... frame, 304 ... information processing unit, 401 ... CPU, 402 ... ROM, 403 ... RAM, 404 ... non-volatile storage, 405 ... RTC, 406 ... Wireless LAN interface, 407 ... Wireless communication unit, 408 ... Bus, 409 ... Serial interface, 410 ... LED, 411 ... Speaker, 412 ... Imaging device, 501 ... Face detection processing unit, 502 ... Feature point extraction unit, 503 ... Vector analysis unit, 504 ... Engagement calculation unit, 505 ... Pulse detection area extraction 506 ... Pulse calculation unit, 507 ... Input / output control unit, 508 ... Server access information, 509 ... Interface selection unit, 601 ... Subject, 801 ... CPU, 802 ... ROM, 803 ... RAM, 804 ... Non-volatile storage, 805 ... RTC, 806 ... NIC, 807 ... bus, 901 ... input / output control unit, 902 ... authentication processing unit, 903 ... user master, 904 ... log table, 905 ... expression estimation unit, 1101 ... CPU, 1102 ... ROM, 1103 ... RAM, 1104 ... display unit, 1105 ... operation unit, 1106 ... nonvolatile storage, 1107 ... RTC, 1109 ... wireless communication unit, 1110 ... bus, 1201 ... input / output control unit, 1202 ... server access information, 1203 ... interface selection , 1204 ... log table, 1205 ... alarm transmission processing

Claims (8)

A mirror-type terminal that captures the face of the person being monitored and outputs predetermined data;
A server that receives the predetermined data from the mirror terminal;
A single person monitoring system comprising a monitoring terminal that receives data output from the mirror type terminal and transferred from the server;
The mirror type terminal is
A mirror that can reflect the face of the person being monitored,
An imaging device that is incorporated in the mirror and captures the face of the person to be monitored and outputs image data;
A face detection processing unit for detecting the face of the monitoring subject included in the image data;
A feature point extracting unit that generates feature point data including face feature points from the face extraction image data obtained by extracting the face of the monitoring target, output by the face detection processing unit;
An input / output control unit connected to a network and transmitting the feature point data to the monitoring terminal via the server;
The server
An input / output control unit that is connected to the mirror terminal and the monitoring terminal through the network and transfers the feature point data transmitted from the mirror terminal to the monitoring terminal;
The monitoring terminal is:
An input / output control unit for receiving the feature point data transferred from the server;
A single person monitoring system comprising a display unit for displaying the feature point data. The mirror type terminal further includes
Based on the feature point data, a pulse detection region that performs extraction of image data corresponding to a part of the face of the monitoring subject included in the face extraction image data and outputs the obtained partial image data An extractor;
A pulse calculation unit that calculates the pulse of the monitoring subject from the amount of variation on the time axis of the luminance of the specific color component of the partial image data;
The input / output control unit of the mirror type terminal transmits the pulse data output from the pulse calculation unit to the monitoring terminal via the server, in addition to the feature point data.
The input / output control unit of the monitoring terminal receives the pulse data in addition to the feature point data transferred from the server,
The display unit of the monitoring terminal displays the pulse data in addition to the feature point data.
The single person monitoring system according to claim 1. The server further includes:
A facial expression estimation unit that estimates the facial expression of the person to be monitored based on the feature point data transmitted from the mirror-type terminal;
The input / output control unit of the monitoring terminal receives the facial expression estimation result output by the facial expression estimation unit in addition to the feature point data and the pulse data transferred from the server,
The display unit of the monitoring terminal displays the facial expression estimation result in addition to the feature point data and the pulse data.
The single person monitoring system according to claim 2. The mirror type terminal further includes
A vector analysis unit for generating a face direction vector indicating a direction of the face of the monitoring target person and a line-of-sight direction vector indicating the direction of the line of sight of the face of the monitoring target person from the feature point data;
The face direction vector and the line-of-sight direction vector are added to calculate a gaze direction vector indicating where in the three-dimensional space the target person is gazing, and the gaze direction vector is suitable for a predetermined event. An engagement calculating unit that calculates a moving average of the determination results and outputs an engagement value.
The input / output control unit of the mirror type terminal transmits the engagement value output from the engagement calculation unit to the monitoring terminal via the server in addition to the feature point data and the pulse data. ,
The input / output controller of the monitoring terminal receives the engagement value in addition to the feature point data and the pulse data transferred from the server,
In addition to the feature point data and the pulse data, the display unit of the monitoring terminal displays the engagement value.
The single person monitoring system according to claim 2. A mirror that can reflect the face of the person being monitored,
An imaging device that is incorporated in the mirror and captures the face of the person to be monitored and outputs image data;
A face detection processing unit for detecting the face of the monitoring subject included in the image data;
A feature point extracting unit that generates feature point data including face feature points from the face extraction image data obtained by extracting the face of the monitoring target, output by the face detection processing unit;
A mirror-type terminal that is connected to a network and includes an input / output control unit that transmits the feature point data to a predetermined monitoring terminal. Furthermore,
Based on the feature point data, a pulse detection region that performs extraction of image data corresponding to a part of the face of the monitoring subject included in the face extraction image data and outputs the obtained partial image data An extractor;
A pulse calculation unit that calculates the pulse of the monitoring subject from the amount of variation on the time axis of the luminance of the specific color component of the partial image data;
The input / output control unit transmits pulse data output from the pulse calculation unit to the monitoring terminal in addition to the feature point data.
The mirror type terminal according to claim 5. Furthermore,
A facial expression estimation unit that estimates the facial expression of the monitoring subject based on the feature point data;
The input / output control unit transmits the facial expression estimation result output from the facial expression estimation unit to the monitoring terminal in addition to the feature point data and the pulse data.
The mirror type terminal according to claim 6. Furthermore,
A vector analysis unit for generating a face direction vector indicating a direction of the face of the monitoring target person and a line-of-sight direction vector indicating the direction of the line of sight of the face of the monitoring target person from the feature point data;
The face direction vector and the line-of-sight direction vector are added to calculate a gaze direction vector indicating where in the three-dimensional space the target person is gazing, and the gaze direction vector is suitable for a predetermined event. An engagement calculating unit that calculates a moving average of the determination results and outputs an engagement value.
The input / output control unit transmits the engagement value output by the engagement calculation unit to the monitoring terminal in addition to the feature point data and the pulse data.
The mirror type terminal according to claim 6.

Images