JP2022179438A - Non-contact type state detection device and non-contact type state detection program - Google Patents

Abstract

To provide a non-contact type state detection device capable of grasping a health state and a psychological state of an object person in a remote place over a screen by acquiring biological information on the object person from moving image information captured in a non-contact manner, and estimating a state of the object person, and capable of managing the health state and the psychological state easily.SOLUTION: A non-contact type state detection device includes: a measurement area extraction unit 32 for specifying a measurement area for measuring biological information from moving image information in which an object person 11 is imaged; a detection unit 34 for detecting a pulse wave of the object person 11 on the basis of a change in a luminance value between the frames of the moving image information in a skin area of the measurement area; and a state estimation unit 35 for estimating the state of the object person 11 on the basis of a result of the detection by the detection unit 34. As needed, respiration of the object person 11 is detected on the basis of a change in a vertical position of a shoulder area detected between the frames of the moving image information, and a change in a facial expression of the object person 11 is detected on the basis of a change in a predetermined feature point detected between the frames of the moving image information in a face area.SELECTED DRAWING: Figure 2

Description

TECHNICAL FIELD The present invention relates to a non-contact state detection device for detecting biometric information without contact, and more particularly to a non-contact state detection device for estimating the state of a subject from acquired biometric information.

A technique is known for estimating the state of a subject, such as stress and emotion, based on biological information or the like acquired from the subject. The technique disclosed in Patent Literature 1 captures images of a person to be measured with a camera before and after stress is applied when the person to be measured is in a healthy state, and captures the body of the person to be measured. Feature points are extracted from the image data, the coordinates of these feature points are stored in a standard database, and the coordinates of the feature points when the person to be measured is in a healthy state stored in the standard database and the coordinates of the feature points at the time of measurement are stored in the standard database. The coordinate difference is represented by a vector, and the stress that the subject receives during the measurement is evaluated from this vector.

Further, in the technology disclosed in Patent Document 2, the input interface receives a first biomedical signal corresponding to the pulse of the subject and a second biomedical signal corresponding to the electroencephalogram of the subject, and the processor responds to the first biomedical signal. a first data set including a first value corresponding to the second biosignal and a second data set including a plurality of state values corresponding to a plurality of state quantities arranged on the Russell's torus model a second data set, the storage storing colors associated with each of the plurality of states, and the display device displaying a first coordinate axis corresponding to the first biomedical signal and a second data set corresponding to the second biomedical signal; A coordinate plane formed by two coordinate axes is displayed, and the processor causes the indicator to be displayed at a position on the coordinate plane defined by the first data set and associated with one of the plurality of state values having the maximum value. The color is displayed on the display device.

On the other hand, there is known a technology that expands and visualizes minute changes that cannot be perceived by human vision using video data. With these technologies, it is possible to visualize the pulse by magnifying slight skin color changes due to changes in blood flow, and to visualize the vibration state of the crane by magnifying the minute vibrations of the crane. (See Non-Patent Document 1).

JP 2017-176762 A Japanese Unexamined Patent Application Publication No. 2020-185138

Massachusetts Institute of Technology, “Video Magnification”, [online], last updated June 2015, [searched February 12, 2021], Internet <URL: http://people.csail.mit.edu/mrub/vidmag/ ＞

The technique disclosed in Patent Document 1 extracts predetermined feature points before and after stress is applied, and evaluates the stress from the coordinate vectors. There is a problem that it has to be done and it takes time and effort.

Also, it is described that a heartbeat is obtained from a finger or a chest near the heart, but in that case a contact-type sensor is required to obtain the heartbeat. For example, some subjects with mental disabilities are extremely reluctant to have sensors come into contact with their bodies. , there is a problem that it becomes difficult to use the technique shown in Patent Document 1.

The technique disclosed in Patent Document 2 is a technique for estimating a state from a pulse and an electroencephalogram, but it may be difficult to apply it for the same reason as above because it is necessary to attach a sensor to the body in order to obtain the pulse and an electroencephalogram. have the problem of obtaining

Although the technique shown in Non-Patent Document 1 realizes visualization of the pulse and the like by photographing with a camera in a non-contact state, it cannot determine the state of the subject's emotions, stress, and the like.

The present invention acquires the biological information of a subject from video information captured without contact and estimates the state of the subject, thereby grasping the health and psychological state of the subject at a remote location through the screen. Provided is a non-contact state detection device that can be easily managed by

A non-contact state detection device according to the present invention includes measurement area specifying means for specifying a measurement area for measuring biological information from moving image information of a subject, and moving image information in the skin area of the measurement area. detecting means for detecting the subject's pulse wave based on the change in luminance value between frames; and state estimating means for estimating the state of the subject based on the detection result of the detecting means. .

As described above, in the non-contact state detection device according to the present invention, a measurement area for measuring biological information from moving image information of a subject is specified, and in the skin area of the measurement area, the moving image information The pulse wave of the subject is detected based on the change in luminance value between frames, and the state of the subject is estimated based on the detection results. It is possible to estimate the subject's condition, and it is possible to grasp the subject's health condition and psychological condition.

In the non-contact state detection device according to the present invention, the detection means detects the object on the basis of the change in the vertical position of the shoulder region detected between the frames of the moving image information in the shoulder region of the measurement region. It detects a person's breathing.

As described above, in the non-contact state detection device according to the present invention, in the shoulder region of the measurement region, respiration of the subject is detected based on the change in the vertical position of the shoulder region detected between the frames of the moving image information. Since the detection is performed, it is possible to more accurately estimate the subject's condition from the pulse wave and respiration information.

In addition, since there is a correlation between the change in the pulse wave due to the change in the condition and the change in the respiration due to the change in the condition, even if the pulse wave information cannot be obtained, it can be supplemented with respiration. It has the effect of being able to

In the non-contact state detection device according to the present invention, the detection means changes the facial expression of the subject based on changes in predetermined feature points detected between frames of the moving image information in the face area of the measurement area. It detects changes.

As described above, in the non-contact state detection device according to the present invention, in the face area of the measurement area, the facial expression of the subject changes based on the change in the predetermined feature points detected between the frames of the moving image information. is detected, there is an effect that it becomes possible to more accurately estimate the subject's condition from the information on the pulse wave and changes in facial expression.

In the non-contact state detection device according to the present invention, the detection means expands minute changes between frames by vector amplification to detect the changes.

As described above, in the non-contact state detection device according to the present invention, since the detection means expands minute changes between frames by vector amplification to detect the changes, minute changes in the skin area due to blood flow are detected. , fine up-and-down vibrations of the shoulder region due to respiration, fine changes in facial expressions, etc. can be expanded and perceived as clear changes.

A non-contact type state detection device according to the present invention comprises: a motion information storage unit for storing information on a proper motion according to a state; and a display control means for displaying the extracted action information on a display.

As described above, in the non-contact state detection device according to the present invention, the motion information storage means for storing the information of the proper motion according to the state and the proper motion information according to the state estimated by the state estimation means are stored. Since the motion information extraction means for extracting from the motion information storage means and the display control means for displaying the extracted motion information on a display are provided, biofeedback according to the estimated state of the subject is possible, and the subject It has the effect of stabilizing the mental state of the person.

In the non-contact state detection device according to the present invention, the motion information stored in the motion information storage means includes parasympathetic nerve dominant motion information for making the subject's parasympathetic nerves dominant, and It is divided into sympathetic nerve dominant action information for making the sympathetic nerve dominant.

Thus, in the non-contact state detection device according to the present invention, the motion information stored in the motion information storage means is the parasympathetic nerve dominant motion information for making the subject's parasympathetic nerves dominant. , and sympathetic nerve dominant action information for making the subject's sympathetic nerves dominant. By presenting motion information suitable for each situation, such as when the subject wants to calm down (state of parasympathetic nerve dominance), it is possible to control the subject's feelings so that he/she can adapt to various situations. .

In the non-contact state detection device according to the present invention, the motion information stored in the motion information storage means includes long-term motion information indicating proper motion for a long-term state, and long-term motion information for a short-term state. and short-term operation information, which is appropriate operation.

As described above, in the non-contact state detection device according to the present embodiment, the motion information stored in the motion information storage means includes long-term motion information, which is appropriate motion for a long-term state, and short-term motion information. Since it is divided into short-term action information, which is appropriate action for a serious condition, for example, suitable action for a condition that requires long-term treatment such as psychosomatic disease and excessive short-term action information By classifying and coping with the coping action suitable for stress, there is an effect that more appropriate coping becomes possible.

A non-contact state detection device according to the present invention includes temperature information acquiring means for acquiring temperature information of the subject imaged by a thermal sensor, and face extraction for extracting the facial region of the subject based on the temperature information. wherein the measurement area specifying means specifies the measurement area by superimposing the face area extracted by the face extraction means on the moving image information.

As described above, in the non-contact state detection device according to the present embodiment, the temperature information acquisition means acquires the temperature information of the subject imaged by the thermal sensor, and the face of the subject is detected based on the temperature information. face extraction means for extracting an area, and the measurement area specifying means specifies the measurement area by superimposing the face area extracted by the face extraction means on the moving image information. For example, there is an effect that it is possible to easily extract from the temperature distribution and lighten the processing without performing complicated processing such as image recognition using AI.

1 is a system configuration diagram of a non-contact state detection system using the non-contact state detection device according to the first embodiment; FIG. 1 is a functional block diagram showing the configuration of a state detection device according to a first embodiment; FIG. FIG. 4 is an image diagram showing processing when a pulse wave is detected by the state detection device according to the first embodiment; FIG. 4 is an image diagram showing processing in the case of detecting respiratory information in the state detection device according to the first embodiment; FIG. 11 is a first diagram showing an example of a moving image when vector expansion is performed in the state detection device according to the first embodiment; FIG. 7 is a second diagram showing an example of a moving image when vector extension is performed by the state detection device according to the first embodiment; FIG. 11 is a third diagram showing an example of a moving image when vector expansion is performed in the state detection device according to the first embodiment; It is a figure which shows an example of the Lorenz plot in the estimation process of the state detection apparatus which concerns on 1st Embodiment. 4 is a flowchart showing processing of the state detection device according to the first embodiment; It is a functional block diagram which shows the structure of the state detection apparatus which concerns on 2nd Embodiment. 9 is a flowchart showing processing of the state detection device according to the second embodiment; It is a figure which shows the structure of the sensor part in the state detection apparatus which concerns on 3rd Embodiment. FIG. 11 is a diagram showing processing for identifying a measurement region using a thermal sensor in the state detection device according to the third embodiment; It is a figure which shows the process outline at the time of using the state detection apparatus which concerns on 4th Embodiment in a workplace. It is a figure which shows the process outline at the time of using the state detection apparatus which concerns on 4th Embodiment in the educational site. FIG. 16 is a diagram showing an example of a screen displayed on the administrator terminal when processing is performed in FIG. 14 or FIG. 15;

(First embodiment of the present invention)
A non-contact state detection device according to this embodiment will be described with reference to FIGS. 1 to 9. FIG. The non-contact state detection device according to the present embodiment detects the biological information of a subject in a non-contact manner, and estimates and manages the state of the subject from the detected biological information.

FIG. 1 is a system configuration diagram of a non-contact state detection system using a non-contact state detection device according to this embodiment. The non-contact state detection system 1 includes a camera 12 that captures an image of a subject 11, and image information of the subject 11 with whom the subject 11 is having a conversation through an electric communication line such as the Internet, or the subject 11 himself/herself imaged by the camera 12. and a subject terminal 10 having at least a display 13 for displaying.

In addition, the non-contact state detection system 1 includes a camera 22 that captures an image of an administrator 21 who manages the entire system, and a display 23 that displays image information of a target person 11 who is interacting through an electric communication line such as the Internet. An administrator terminal 20 having at least.

Further, the non-contact state detection system 1 receives imaging information of the subject 11 captured by the camera 12 of the subject terminal 10, and performs state estimation processing of the subject 11 using the received imaging information. A non-contact state detection device 30 (hereinafter abbreviated as state detection device 30) is provided for transmitting various information including the processing results to each subject terminal 10 and administrator terminal 20. FIG.

Note that the administrator terminal 20 in FIG. 1 is not an essential component, and the target person 11 does not necessarily have to interact with the administrator 21 through the display 13. However, the description will be made on the assumption that the biological information of the subject 11 is obtained or the state is estimated.

The camera 12 of the target person terminal 10 captures a moving image of the target person 11 while the target person 11 and the administrator 21 are having a one-to-one conversation, and information about the captured moving image is transmitted to the state detection device 30. be. The state detection device 30 detects the biological information of the subject 11 from the information of the moving image transmitted from the subject terminal 10 . The details of the process of detecting biometric information from moving image information will be described later.

The state detection device 30 estimates the state of the subject 11 from the detected biological information. The state of the subject 11 is, for example, a state of excessive stress, a relaxed state, an uneasy state, an excited state, or the like, which includes mental elements. These states are managed daily as the health condition of the subject 11 and used for health management.

When the manager 21 and the target person 11 are able to interact with each other, the state detection device 30 transmits the biometric information and the state information of the target person 11 who is the conversation partner to the manager terminal 20, and the information is sent to the manager terminal 20. It can be displayed on the display 23 . The administrator 21 can interact with the subject 11 while referring to the subject's 11 biological information, mental state, etc. displayed on the display 23 .

The relationship between the administrator 21 and the target person 11 corresponds to, for example, a boss and a subordinate, a teacher and a student, an employer and a worker, and an organizer and a participant. It is possible to manage daily management (especially mental health management) while interviewing. In particular, remote communication using electric communication lines such as telework is expanding, and the frequency of use of the non-contact state detection device according to this embodiment is increasing as the opportunity to image the target person 11 with a camera increases. As a result, it becomes possible to easily manage the health of the subject 11 .

In addition to the above, for example, it can also be applied to job interviews and arranged marriages conducted through electric communication lines, and it is possible to respond while considering the feelings of the other party.

FIG. 2 is a functional block diagram showing the configuration of the state detection device 30 according to this embodiment. The state detection device 30 includes an input unit 31 that receives moving image information of the subject 11 captured by the camera 12 of the subject terminal 10, and information about the area necessary for measuring the biological information of the subject 11, the biometric information, and so on. Based on the information in the basic information storage unit 33 that stores basic information such as information necessary for information detection and/or parameters necessary for estimation calculation, biometric information is extracted from the input moving image information of the subject 11. a measurement region extraction unit 32 for extracting a measurement region necessary for measuring the measurement region, a detection unit 34 for detecting the biological information of the subject 11 from changes between frames of the moving image information of the extracted measurement region, and the detected a state estimating unit 35 for estimating the state of the subject 11 based on the biological information obtained; a state information storage unit 36 for storing the estimated state and biological information of the subject 11; An output control unit 37 for outputting to the subject terminal 10 and the administrator terminal 20 is provided.

In this embodiment, non-contact detectable biological information includes, for example, pulse wave, respiration rate, facial expression, body temperature, etc. At least one or more of these information is detected from moving image information. A measurement area to be extracted from the moving image information is set according to the biometric information to be detected, and the information is stored in the basic information storage unit 33 . The measurement region extraction unit 32 extracts a measurement region to be extracted based on the information in the basic information storage unit 33 according to the type of biological information to be detected.

Specifically, for example, when detecting pulse wave information, it is necessary to capture minute changes in the luminance value of the skin surface caused by blood flow, so at least the skin region must be extracted as the measurement region. Specifically, when the image is captured by the camera 12, the face area of the subject 11 that is highly likely to have exposed skin is extracted. The areas of the face that are particularly easy to extract are the cheeks and forehead. Since it may be difficult to extract the skin area, it is desirable to extract both areas and then extract the skin area at either or both locations.

In addition, when detecting the respiration rate, it is necessary to capture vertical microvibrations of the shoulder, so the shoulder region of the subject 11 is extracted as the measurement region. When detecting facial expressions, the face area is extracted as the measurement area. When detecting body temperature, the skin area is extracted as the measurement area. As for body temperature, it is necessary to use a thermal sensor capable of measuring body surface temperature instead of a normal camera.

From the above, it is sufficient to extract the region above the shoulder as the measurement region for detecting the pulse wave, respiration rate, facial expression, body temperature, etc., and it is not necessary to image the whole body or the back of the subject 11. Instead, the biometric information can be detected simply by sitting in front of the camera 12 , that is, the subject terminal 10 .

The detection unit 34 extracts the biological information from the change between frames of the moving image in the measurement area extracted by the measurement area extraction unit 32 . FIG. 3 is an image diagram showing processing when detecting a pulse wave. Here, a luminance value is obtained by extracting a measurement region (here, the cheek portion in the skin region) as shown in FIG. 3B from each frame in the moving image information as shown in FIG. , as shown in FIG. 3(C), shows the processing from the change in luminance value to the detection of the waveform of the pulse wave. As shown in FIG. 3, it not only detects the pulse rate by capturing changes in luminance value, but also obtains pulse wave information (peak interval, pulse intensity, etc.) as a waveform as shown in FIG. 3(C). detection becomes possible. Such pulse wave information is used by the state estimating section 35 at the next stage, and is also stored in the state information storage section 36 to be used as information for health management.

Moreover, the detection unit 34 detects the respiration rate as necessary. FIG. 4 is an image diagram showing processing when detecting respiration information. As described above, the respiration rate is detected by capturing changes in the movement of the shoulder region that slightly vibrates up and down according to the respiration of the subject 11 . 4A shows moving image information, FIG. 4B shows processing for detecting changes in vertical movement of the shoulder region, which is the measurement region, and FIG. 4C shows the vertical movement of the shoulder region. It shows the process of detecting a respiratory waveform from changes. In this case as well, it is possible to detect not only the respiration rate but also the respiration interval (normal respiration/deep respiration/rough respiration, etc.) and intensity as a waveform as shown in FIG. Such respiration information can be used by the state estimation unit 35 at the next stage, and may be stored in the state information storage unit 36 and used as information for health management.

Further, the detection unit 34 detects facial expressions as necessary. Facial expressions can be detected according to, for example, changes in the corners of the eyes up and down and changes in the corners of the mouth up, down, left, and right, and generally known facial expression detection techniques can be used. The body temperature can also be detected by capturing an image of the subject 11 with a generally known infrared thermal sensor. Such facial expression and body temperature information can be used by the state estimation unit 35 at the next stage, and may be stored in the state information storage unit 36 and used as information for health management.

Note that the detection unit 34 may use a technique of vector extension to more reliably detect biometric information. 5 to 7 are diagrams showing an example of a moving image in the case of vector extension. FIG. 5 shows an expanded change in the brightness value of the skin area, FIG. 6 shows an expanded change in the vertical movement of the shoulder area, and FIG. 7 shows an expanded change in facial expression. indicates a moving image without vector amplification, and (B) indicates a moving image with vector amplification.

In the case of FIG. 5, by expanding the change in luminance value between frames in the moving image information to increase the degree of color change, minute color changes on the skin surface due to the pulse that cannot be recognized without vector amplification can be recognizably displayed. In the case of FIG. 6, by expanding and increasing the change in the vertical movement of the shoulder region between frames in the moving image information, it is possible to recognize the vertical vibration of the shoulder region due to respiration, which is almost unrecognizable without vector amplification. status can be displayed. In the case of FIG. 7, by expanding the change in facial expression between frames in the moving image information, even a slight change in facial expression can be exaggerated. 5 to 7 show, as an example, an image of how the actual change in the subject 11 is exaggerated by comparing the case where vector amplification is not performed and the case where vector amplification is performed. , and the method of exaggeration by vector amplification can be appropriately set.

Regarding the change in facial expression in FIG. 7, for the target person 11 whose facial expression hardly changes in response to emotions, especially those seen in mentally handicapped persons, exaggerated changes make it easier to estimate the emotion. . Furthermore, by exaggerating the change in the moving image, it becomes possible to easily confirm the change in the biometric information visually as shown in FIGS. 5 to 7 .

The state estimation unit 35 estimates the state of the subject 11 from the biological information detected by the detection unit 34 . For example, the autonomic nervous system is analyzed by Lorenz plot from the detected pulse wave information. More specifically, for example, as shown in FIG. 8, the value of L (long axis length)/T (short axis length) is calculated from the Lorenz plot, and the emotion is determined according to this L/T. Note that the table showing the relationship between the L/T and the state of the subject 11 shown in FIG. Here, it is set to determine an irritated state, a tense state, a calm state, and a relaxed state. In addition to this, for example, determination of sympathetic nerve dominance/parasympathetic nerve dominance, determination of concentration state/distraction state, and the like can be performed. Information about the estimated condition of the subject 11 is stored in the condition information storage unit 36 and can be used as information for health management.

If information about breathing is detected, a Lorentzian plot using the breathing information may be created to estimate the state of the subject 11, or if the pulse wave cannot be detected, the Lorentzian plot using the breathing information A plot may be created. Further, when both pulse wave information and respiration information can be detected, only one of the information may be used according to a preset priority, or a table as shown in FIG. 8B may be used. may be prepared for L/T (pulse wave)+L/T (respiration), and the relationship between L/T (pulse wave)+L/T (respiration) and the state may be obtained.

Furthermore, if information related to facial expressions is detected, the state may be estimated by quantifying the facial expressions. For example, the strength of a smile or an angry face is quantified according to the degree of raising and lowering the corners of the eyes and the degree of the spread of the mouth (for example, from an angry state to a happy state is digitized by -10 to 10), and normalized to the above. By reflecting on L/T, the state of the subject 11 can be estimated more accurately. Similarly, if the body temperature is high, it is in a tense state, and if it is low, it is quantified as a relaxed state. can do.

In the above description, information such as pulse wave, respiration, facial expression, and body temperature is normalized into one parameter. good. For example, by estimating "tensed state" to "relaxed state" from pulse wave and breathing, estimating "emotions" from facial expressions, and estimating "excited state" to "normal state" from body temperature. , the state may be estimated by multiple indices.

The output control unit 37 transmits the biological information detected by the detection unit 34 and the state of the subject 11 estimated by the state estimation unit 35 to the subject terminal 10 and the administrator terminal 20 . The biological information and state information transmitted to the subject terminal 10 and the administrator terminal 20 are displayed on the displays of the respective terminals and referred to by the subject 11 and the administrator 21 .

When the administrator 21 remotely interacts with the target person 11 through the display, the administrator 21 can interact while confirming the target person's biological information and state in real time. It is possible to respond. For example, when it is determined that the target person 11 is in an excited state, it is possible to take measures such that the target person 11 can transition to a relaxed state while speaking as slowly as possible. In addition, the subject 11 can grasp his/her own condition by confirming his/her biological information and condition, and can take conscious actions so as to stabilize his/her mental condition.

In addition, the administrator 20 periodically (for example, every six months or every year) confirms the biological information and the state information of the subject 11 stored in the state information storage unit 36, thereby managing the health of the subject 11. It is possible to do

Next, operation of the state detection device will be described. FIG. 9 is a flow chart showing the operation of the state detection device according to this embodiment. First, the input unit 31 inputs and acquires moving image information of the target person 11 transmitted from the target person terminal 10 (S1). The measurement area extracting unit 32 extracts a face area (especially a skin area) and a shoulder area as measurement areas from the acquired moving image information (S2). The detection unit 34 detects biological information such as pulse wave, respiration, facial expression, and/or body temperature from the frame-to-frame changes in the moving image information in the extracted measurement area and from the thermal sensor (S3). The state estimation unit 35 estimates the state of the subject 11 from the detected biological information (S4). The detected biological information and the status information of the subject 11 are output (transmitted) to the administrator terminal 20 and the subject terminal 10 (S5), and the process ends.

As described above, in the condition detection device according to the present embodiment, a measurement area for measuring biological information is specified from the moving image information of the subject 11, and the moving image information is detected in the skin area of the measurement area. The pulse wave of the target person 11 is detected based on the change in luminance value between frames, and the condition of the target person is estimated based on the detection results. , the condition of the subject 11 can be easily estimated, and the health condition and psychological state of the subject 11 can be grasped.

In addition, in the shoulder region of the measurement region, since the respiration of the subject 11 is detected based on the change in the vertical position of the shoulder region detected between the frames of the moving image information, the subject's 11 pulse wave and respiration information state can be estimated more accurately. Furthermore, since there is a certain degree of correlation between the change in pulse wave with respect to the change in the state of the subject 11 and the change in respiration with respect to the change in state, even if a state occurs in which pulse wave information cannot be obtained, It can be complemented by breathing.

Furthermore, in the face area of the measurement area, information on changes in pulse wave and facial expression is detected based on changes in predetermined feature points detected between frames of the moving image information. It becomes possible to more accurately estimate the state of the subject 11 from the above.

Furthermore, since the detection unit 34 expands minute changes between frames by vector amplification to detect the changes, minute changes in the skin area due to blood flow, minute vertical vibrations in the shoulder area due to respiration, and minute facial expressions are detected. changes can be exaggerated and captured as clear changes.

In addition to the pulse wave, respiration, facial expression, and body temperature, it is possible to detect the state of the autonomic nervous system by, for example, the degree of pupil dilation. In this case, by extracting the image of the pupil and detecting the size of the pupil, it is possible to estimate the state of sympathetic nerve dominance/parasympathetic nerve dominance. Then, similarly to the above, quantification according to the size of the pupil (change from the normal state in daily life) may be performed, normalized, and reflected in the L/T.

(Second embodiment of the present invention)
A non-contact state detection device according to this embodiment will be described with reference to FIGS. 10 and 11. FIG. The non-contact state detection device according to the present embodiment extracts appropriate countermeasures and presents them to the subject 11 according to the state of the subject 11 estimated based on the biological information. In this embodiment, explanations that duplicate those of the first embodiment will be omitted.

The subject 11 includes, for example, a healthy person, a disabled person, a child, an elderly person, a person with symptoms of a disease, a healthy person, a person with a mental illness, a motivated person, a unmotivated person, and a nervous person. It is possible to become a person with various characteristics, such as an easy-going person or a calm person. Among these, people in a relatively stable state do not need to take immediate action, but it is desirable for people in an unstable state to be able to shift to emotions and autonomic nerves that stabilize the state as much as possible. In addition, even if there are no symptoms, especially mentally or physically, at the present time, it is possible to maintain good health for a long time by stabilizing the mental state and autonomic nerves on a daily basis. In this embodiment, by specifically presenting a coping method for stabilizing the mental state and the autonomic nerve state to the subject 11, the mental state and the autonomic nerve state are stabilized.

FIG. 10 is a functional block diagram showing the configuration of the state detection device 30 according to this embodiment. What is different from the case of FIG. 2 in the first embodiment is that a coping information storage unit 81 that stores information about a coping method according to the state of the subject 11 and an appropriate and a coping information extracting unit 82 for extracting coping methods from the coping information storage unit 81 .

The coping information stored in the coping information storage unit 81 includes sympathetic nerve dominant action information for causing the subject 11 to perform an action for making the sympathetic nerve dominant according to the state of the subject 11; It has parasympathetic nerve dominant action information for causing the subject to perform the action for shifting the parasympathetic nerve to the dominant state, and appropriate coping information is extracted according to the state of the subject 11 . For example, when the state estimating unit 35 needs to relax the subject 11 because the subject 11 is in an extremely excited state (for example, when the subject 11 is overworked and has a high degree of tension or is under stress in human relationships), Parasympathetic nerve dominant motion information is extracted, and conversely, when the subject is in an extremely relaxed state and the degree of tension needs to be increased immediately (for example, when the subject is too relaxed in a room), the sympathetic nerve dominant motion information is extracted. be done.

Movements to shift to sympathetic nerve dominance include, for example, movements that inevitably increase the breathing rate and movements that increase perspiration. Goo), do gymnastics, watch exciting videos, etc. On the other hand, as an action to shift to parasympathetic nerve dominance, for example, there is an action to slow down the breathing rate and relax. , massage, reading, listening to classical music, deep breathing/abdominal breathing, chewing gum, etc.

In general, it is often said that it is preferable to shift to a parasympathetic nervous system that is in a relaxed state, but if either of the sympathetic nervous system or the parasympathetic nervous system becomes too dominant, it will cause poor physical condition, etc. It should be well kept. For example, if going out is restricted due to the issuance of a state of emergency, etc., and people are forced to stay at home all day long, they may feel relaxed enough but still feel unwell. . In other words, after distinguishing between sympathetic nerve-dominant motion information and parasympathetic nerve-dominant motion information as described above, by presenting coping information suitable for each, it is possible to maintain the balance of the autonomic nerves.

Further, the coping information stored in the coping information storage unit 81 includes long-term action information for causing the subject 11 to perform an action for long-term coping according to the state of the subject 11, and short-term action information. and short-term action information for causing the target person 11 to perform an action for coping with the problem, and appropriate coping information is extracted according to the condition of the target person 11 . For example, if the state estimating unit 35 develops a state that the subject 11 needs to deal with in the long term (for example, psychosomatic disease (depressive state, headache, atopic dermatitis, irritable bowel syndrome, etc.) due to accumulation of stress, etc.) If it is estimated that the driver is in a state where the driver is in a state where the worker is in a state where the worker is in a state of being overwhelmed, long-term operation information is extracted, and conversely, a state that needs to be dealt with in the short term. When it is estimated that the person is in an extremely tense state, etc., short-term motion information is extracted.

Examples of long-term movements include those that are habitually performed in daily life and that are particularly useful for improving one's physical condition. ), drinking water on a regular basis, correcting your posture, spending time with a smile, eating dinner 3 hours before bedtime, taking a slow bath at a suitable temperature, and tidying up your room. On the other hand, short-term motions include, for example, motions that can be executed immediately on the spot and do not take much space or time. In addition to the actions of , other actions include solving difficult problems in a short time, solving simple problems without stress, looking up at the sky, and seeing exciting/relaxing colors.

Furthermore, long-term movements and short-term movements are further subdivided into long-term movements in which the sympathetic nerve is dominant, long-term movements in which the parasympathetic nerve is dominant, short-term movements in which the sympathetic nerve is dominant, and short-term movements in which the parasympathetic nerve is dominant. It may be divided into actions. By dividing in this way, it is possible to present coping information more suitable for the condition of the target person 11 .

In the coping information storage unit 81, the above-described motions that make the sympathetic nerves dominant, motions that make the parasympathetic nerves dominant, motions for long-term coping, and motions for short-term coping are stored for the subject 11. Information for execution is stored. That is, for example, when it is desired to perform an action of exerting force or releasing force, an announcement to that effect is made by text or voice, and a recommended execution time is presented. In addition, images/music that excites/relaxes, goo/pah images and sounds, animations and sounds that make people smile, counter images and sounds that correspond to the appropriate breathing rate, and other actions that accompany each of the above-mentioned actions. Images and sounds are stored, and based on these, a suitable execution method (imitation, viewing, moving/move as instructed, etc.) is extracted.

Each of the countermeasure information extracted by the countermeasure information extraction unit 82 is transmitted to the target person terminal 10 by the output control unit 37 and displayed on the display 13 . The target person 11 actually performs the action while referring to the countermeasure information displayed on the display 13 . For example, when coping information "take a deep breath with a predetermined rhythm" is extracted, display and sound matching the predetermined rhythm are output to the display 13 (speaker in the case of sound) of the target user terminal 10. The subject 11 takes a deep breath according to the sound and the display, thereby making the parasympathetic nerve dominant and shifting to a relaxed state.

If necessary, evaluation means may be provided for evaluating coping actions performed by the subject 11 (in particular, short-term actions that can be confirmed on the spot). For example, when a coping action to make a smile is presented, the smile actually made by the subject 11 may be determined by AI, and whether the smile is sufficient may be evaluated (scored). In addition, it is also possible to evaluate whether or not the subject breathed at the timing as instructed (check the shoulder movement by vector amplification) and whether or not the image was properly viewed (whether the eyes are facing the screen). The target person 11 can receive these evaluation results and perform more appropriate coping operations.

Next, the operation of the state detection device according to this embodiment will be described. FIG. 11 is a flow chart showing the operation of the state detection device according to this embodiment. Since the processing from S1 to S4 is the same as that of FIG. 9 in the first embodiment, description thereof is omitted. When the condition of the target person 11 is estimated in S4, the coping information extraction unit 82 extracts appropriate coping information for the condition of the target person 11 from the coping information storage unit 81 (S5). The output control unit 37 outputs (transmits) the extracted coping information together with the status information to the subject terminal 10 (S6), and ends the process.

As described above, in the state detection device according to the present embodiment, biofeedback corresponding to the estimated state of the subject 11 is possible, and the mental state of the subject 11 can be stabilized. In other words, it is possible to stabilize biological information that is difficult to control intentionally, such as heart rate, body temperature, autonomic nerves, etc. become. In addition, it is possible to prevent various psychosomatic disorders that may occur in the future as much as possible by making the unconscious subject 11 aware of the degree of stress and influence.

In addition, for example, when the user wants to move from the current state to a slightly excited state (state of sympathetic nerve dominance) or when he or she wants to calm down (state of parasympathetic nerve dominance), motion information suitable for each case should be presented. , it becomes possible to control the subject's feelings so that they can adapt to various situations.

Furthermore, by distinguishing between actions suitable for conditions that require long-term coping, such as psychosomatic disorders, and coping actions suitable for excessive stress received in the short term, Appropriate handling becomes possible.

It should be noted that when presenting the countermeasure information on the display 13 of the subject terminal 10, the technique of vector amplification described in the first embodiment may be utilized. Specifically, for example, when making a smile to relax the facial expression is extracted as coping information, the display 13 displays a vector-amplified moving image of a smiling face to serve as a reference, thereby emphasizing the presented content. It is possible to In addition, by displaying a moving image of the target person 11 himself/herself as coping information on the display 13 together with a moving image of a smiling face to be used as a reference, it is possible to display an exaggerated expression by vector-amplifying the facial expression of the subject 11, thereby emphasizing the subject 11. It is possible to execute coping information that is conscious of facial expression changes.

(Third embodiment of the present invention)
A state detection device according to this embodiment will be described with reference to FIGS. 12 and 13. FIG. The condition detection device according to this embodiment reduces the processing load by extracting the measurement region from the subject's temperature information by the measurement region extraction unit 32 . In this embodiment, explanations overlapping those of the above-described embodiments are omitted.

FIG. 12 is a diagram showing the configuration of the sensor unit of the subject terminal 10. As shown in FIG. In the first embodiment, the configuration in which the camera 12 is provided in the target person terminal 10 was described. constitutes the subject terminal 10 as an integral unit. In FIG. 12, the sensor unit 40 includes a camera 41 (having at least the same function as the camera 12) for imaging the subject with visible light, a thermal sensor 42 for detecting temperature (far infrared rays), and a near-infrared imaging device. A near-infrared LED 43 that irradiates light for use, a control unit 44 that controls the operation of each sensor, an input/output interface 45 for inputting and outputting with an external device, and a wired or wireless communication with the external device. and a communication interface 46 for

Note that the subject terminal 10 may be configured only with the sensor section 40 . That is, the subject terminal 10 may be configured as a single device with only the sensor section 40 without input/output devices such as a display and a keyboard. In this case, input/output and communication with other external devices may be performed directly via the input/output interface 45 and the communication interface 46 .

The camera 41 is an image sensor that captures an image of a subject and its surrounding area, and performs imaging using visible light such as sunlight or fluorescent light as a light source, or near-infrared imaging using a near-infrared LED 43 as a light source. When performing imaging using visible light as a light source, a filter that cuts near-infrared light is passed through, and when performing night vision imaging at night, the near-infrared LED 43 is used as a light source and the filter that cuts near-infrared light is removed. Take an image with

The thermal sensor 42 detects far-infrared rays and detects temperature according to the intensity thereof. In general, the stronger the far-infrared rays, the higher the temperature, which is displayed in red, and the weaker the far-infrared rays, the lower the temperature, which is displayed in blue. In the first embodiment, the configuration including the thermal sensor 42 for measuring the body temperature of the subject is described, but in this embodiment, in addition to collecting body temperature information, measurement using the measured temperature information Area identification is performed.

13A and 13B are diagrams showing a process of specifying a measurement region using a thermal sensor in the state detection device according to this embodiment. As shown in FIG. 13A, by imaging the subject with the thermal sensor 42, the body temperature of the subject within a certain range (approximately 35° C. to 37° C., and if necessary, the temperature drop due to the clothes) It is possible to detect the range of 25° C.~), that is, the body temperature range of the subject. On the other hand, as shown in FIG. 13B, the subject is imaged by the camera 41 . By superimposing the body temperature region of FIG. 13A captured by the thermal sensor 42 and FIG. 13B captured by the camera 41, as shown in FIG. It becomes possible to specify the measurement area of the subject. Without such a method, it would be necessary to recognize the target person (human face) by AI, for example, and then extract the measurement area. Since it is enough to just detect the area, the processing can be significantly lightened.

Further, in the present embodiment, in addition to the pulse wave, respiratory rate, facial expression, body temperature, and autonomic nerve state (sympathetic/parasympathetic) described in the first and second embodiments, a plurality of It is possible to detect or obtain biometric information.

Specifically, for example, the target person irradiated with the near-infrared LED 43 is imaged with the camera 41 in a night-vision mode (a mode in which a filter that cuts off near-infrared light is removed), thereby measuring the blood sugar level of the target person. It is possible to measure That is, when the absorbance of the subject's skin portion is measured from the near-infrared image captured by the camera 41, the near-infrared light is absorbed more in the portion with more sugar (glucose), and the black becomes stronger. Therefore, the blood sugar level can be obtained from the difference in gradation between the portion where sugar is detected and the portion where sugar is not detected. Conventionally, it is necessary to actually collect blood to measure the blood sugar level, but by obtaining the investigation value in a non-destructive manner as in this embodiment, it is possible to significantly reduce the physical burden on the subject. becomes. In the measurement of the blood sugar level, it is desirable to image the blood vessels as close to the body surface as possible. may be presented.

Further, for example, the blood pressure of the subject can be calculated from the acquired pulse wave information. Some methods of calculating blood pressure from pulse waves are known, and it can be obtained, for example, as follows. Vascular elastic modulus E is

Here, E ₀ is the vascular elastic modulus at no pressure, a is the correction factor, and P is the blood pressure. From this formula, the blood pressure P is

can be found at As for the vascular modulus, a known value obtained statistically may be used, or it may be obtained from the velocity pulse wave obtained by first-order differentiation of the pulse wave or the acceleration pulse wave obtained by second-order differentiation. good too.

Furthermore, it is also possible to determine, for example, the blood oxygen level of the subject. The near-infrared light emitted from the near-infrared LED 43 has the property of being absorbed by hemoglobin in blood while passing through living tissue. That is, by measuring the concentration of hemoglobin in blood, it is possible to determine the blood oxygen concentration. In order to measure the relative concentrations of oxygenated hemoglobin and deoxygenated hemoglobin, near-infrared light in different wavelength bands may be irradiated. At this time, a plurality of near-infrared LEDs 43 with different wavelengths may be used, or near-infrared light in different wavelength bands may be alternately irradiated using the near-infrared LEDs 43 whose wavelengths are variable. good.

Furthermore, it is also possible, for example, to estimate the degree of concentration of the subject. A correlation between RRI (R-wave interval) and concentration is known, where concentration decreases when RRI rises and concentration remains when RRI is stable. That is, it is possible to estimate the presence or absence of concentration by monitoring the pulse wave.

Collection and transmission of each biometric information in the sensor section 40 are controlled by the control section 44 . For example, when recognizing a target person or acquiring body temperature, the thermal sensor 42 is operated and sensing is performed, and when measuring a blood sugar level or a blood oxygen concentration, or when capturing an image at night time, a nearby sensor is used. While the infrared LED 43 is operated as a light source, the camera 41 takes an image in the night vision mode.

Information acquired by the sensor unit 40 is transmitted to the condition detection device 30 via the communication interface 46, and the condition of the subject is estimated. In the first embodiment, the state estimating unit 35 uses the obtained biological information to estimate the state of the subject by Lorentz plot. In the present embodiment, artificial intelligence ( AI) estimates the information of the subject. In this embodiment, it is possible to acquire or calculate a plurality of types of biometric information as described above in addition to each biometric information in each of the above embodiments. Therefore, the subject's condition is estimated by artificial intelligence (AI) that receives such a large amount of biological information as input. Note that the condition of the subject may be estimated by creating a Lorenz plot unique to each piece of biological information and comprehensively judging them.

For example, a staff member who works at a children's facility, a facility for the disabled, or the like to provide services may be able to determine today's mood and feelings at a glance at each service recipient. Based on the experience of veteran staff, it seems that this is estimated by comprehensively instantaneously judging the atmosphere (for example, appearance, voice, expression, movement, clothing, etc.) emitted by the service recipient. It is very difficult to attach and explain or quantify. Therefore, by using AI that has learned the subject's condition based on various biological information as described above, it is possible to make an estimation similar to that of such a veteran employee. Note that the AI may perform estimation for the entire service target person, or may perform estimation individually for each service target person.

As the state of the subject estimated by the AI function of the state estimating unit 35, in addition to the emotional state described in the first embodiment, for example, the state of health (physical health) determined from each of the above biological information. condition + mental health condition), signs of tantrum, manic-depressive state, degree of fatigue, mood, etc. can be estimated.

(Fourth embodiment of the present invention)
Here, some specific examples will be given to explain how the non-contact state detection device according to the present invention is used. In this embodiment, explanations overlapping those of the above-described embodiments are omitted.

First, as an example, a case where the non-contact state detection system 1 is used in a workplace will be described. By adopting the non-contact state detection system 1 in the workplace, it can be used to improve the workplace environment and productivity as described later. FIG. 14 is a diagram showing an outline of processing when the non-contact state detection system 1 is used in the workplace. In FIG. 14, for workers working in a factory or office, work contents, work places, etc. are set in advance (FIG. 14(A)). The camera 12 of the subject terminal 10 captures an image of the worker's work (FIG. 14(B)). The operator is recognized, identified, and monitored by face recognition (FIG. 14(C)). Information is accumulated while monitoring the state of the worker by the processing described in each of the above embodiments. At the same time, an alert is output when the concentration of the worker is lowered (Fig. 14(D)).

In this way, by outputting an alert when it is estimated that the concentration of the worker has decreased, it is possible to prevent accidents and take safety measures as risk management when performing dangerous work. becomes. In addition, by analyzing the accumulated information, it is possible to find out what kind of work the target worker is doing when the state is stable and concentration is maintained, so it is possible to determine the appropriate work content and It can be useful for allocating workload. Furthermore, since it is known in what kind of work environment the worker's condition is stable and concentration is maintained, it is possible to make use of the improvement of the work environment.

As another example, a case where the non-contact state detection system 1 is used in an educational setting will be described. By adopting the non-contact state detection system 1 in an educational setting, it can be used to improve the classroom environment and improve learning ability, as will be described later. FIG. 15 is a diagram showing an overview of processing when the non-contact state detection system 1 is used in an educational setting. Note that FIG. 15 describes the case of using in a classroom at a school or the like, but it can also be applied to online individual lessons or e-learning. In FIG. 15, learning contents, the location of the classroom, etc. are set in advance for students who take classes in a school classroom (FIG. 15(A)). The camera 12 of the subject terminal 10 takes an image of the student's lesson (FIG. 15(B)). Individual students are recognized, specified, and monitored by face recognition (FIG. 15(C)). Information is accumulated while the student's status is monitored by the processing described in each of the above embodiments. At the same time, an alert is output to the administrator terminal 20 used by the teacher when the student's concentration drops (FIG. 15(D)).

In this way, when it is estimated that a student's concentration is declining, the teacher, who is the administrator, can grasp how much concentration each student has in class. It is possible to realize effective lessons while changing. In addition, by analyzing the accumulated information, it is possible to understand in detail how the biological information and autonomic nerves are changing when the target student is receiving what kind of class from what kind of teacher. Therefore, it is possible to provide detailed support according to how to proceed with learning, class content, good subjects, weak subjects, etc.

In addition, schools can detect changes in the state of students in detail and quickly respond to changes in physical and mental health conditions. By grasping the student's condition, it is possible to implement measures that lead to improvement in grades. In addition, it is possible to draw out latent interests and concerns that the student himself is not aware of, so that he can discover new possibilities for the student and propose classes to take and course selection. Furthermore, since it is possible to know in what kind of classroom environment and at what time period each student is in a stable state and has sustained concentration, it is possible to make use of the improvement of the classroom environment.

Furthermore, by receiving detailed support for students, it is possible to discover possibilities that they themselves are not aware of, and to improve their grades.

FIG. 16 is a diagram showing an example of a screen displayed on the administrator terminal 20 when the processes in FIGS. 14 and 15 are performed. FIG. 16(A) shows a heat map of state changes for each monitored person in the entire monitored area such as workplaces and classrooms, and FIG. 16(B) shows a heat map of state changes over time for each monitored person. showing.

In the heat map of FIG. 16(A), which area of the monitoring target area (workplace, classroom, etc.) is in a stable state, and which area is being monitored. The subject can see at a glance whether the state is unstable. In other words, placement (for example, many people come and go, by the window, near the air conditioner, in the front seat, in the back seat, with certain people nearby, with bosses and teachers nearby, with close friends nearby) etc.), the status of the person to be monitored can be stabilized as much as possible by performing sequential reassignment.

In the heat map of FIG. 16B, it is possible to check the change in state over time for each monitored person in real time, so that appropriate measures can be taken for each monitored person. For example, if concentration is declining, it is recommended to take a break, if it is judged that the accumulated information is unstable due to the work you are not good at, you can change it to the work you are good at. If it is a class, the situation of the person being monitored is stabilized by taking measures such as talking in a relaxed manner.

In this way, by using the non-contact state detection device according to the present embodiment, it can be used to improve the environment of workplaces, schools, etc., or to provide an efficient environment that ensures the safety of workers and students. Detailed monitoring is possible so that work and lessons can be realized.

(Other embodiments)
In each of the above-described embodiments, by performing parallel processing, it is possible to communicate while confirming the other person's biological information through the screen in a state as close to real time as possible. For example, it is possible to detect a pulse wave almost in real time by dividing a frame of moving image information acquired from the subject terminal 10 and performing parallel processing of color extraction for each CPU core. In the system actually developed by the inventors, the pulse wave could be detected within a time lag range of 0.5 seconds to 1.5 seconds.

In this manner, by appropriately dividing frames of moving image information and extracting color and motion information for each CPU core and performing parallel processing, biometric information can be detected in real time and confirmed on the screen.

1 non-contact state detection system 10 subject terminal 11 subject 12 camera 13 display 20 administrator terminal 21 administrator 22 camera 23 display 30 non-contact state detection device (state detection device)
31 input unit 32 measurement region extraction unit 33 basic information storage unit 34 detection unit 35 state estimation unit 36 state information storage unit 37 output control unit 40 sensor unit 41 camera 42 thermal sensor 43 near-infrared LED
44 control unit 45 input/output interface 46 communication interface 81 countermeasure information storage unit 82 countermeasure information extraction unit

Claims (9)

measurement area identifying means for identifying a measurement area for measuring biometric information from moving image information of a subject;
detection means for detecting the subject's pulse wave based on a change in luminance value between frames of the moving image information in the skin region of the measurement region;
and a state estimating means for estimating the state of the subject based on the detection result of the detecting means. In the non-contact state detection device according to claim 1,
A non-contact state detection device in which the detection means detects breathing of the subject based on changes in the vertical position of the shoulder region detected between frames of the moving image information in the shoulder region of the measurement region. . In the non-contact state detection device according to claim 1 or 2,
The non-contact state detection device, wherein the detection means detects a change in facial expression of the subject based on a change in a predetermined feature point detected between frames of the moving image information in the face area of the measurement area. In the non-contact state detection device according to claim 1 or 2,
A non-contact state detection device in which the detecting means expands minute changes between frames by vector amplification to detect the changes. In the non-contact state detection device according to claim 1 or 2,
an operation information storage means for storing information on appropriate operation according to the state;
motion information extraction means for extracting appropriate motion information corresponding to the state estimated by the state estimation means from the motion information storage means;
and display control means for displaying the extracted operation information on a display. In the non-contact state detection device according to claim 5,
The motion information stored in the motion information storage means includes parasympathetic nerve-dominant motion information for making the subject's parasympathetic nerves dominant and sympathetic nerve-dominant motion information for making the subject's sympathetic nerves dominant. A non-contact state detection device that is divided into information and In the non-contact state detection device according to claim 5,
The motion information stored in the motion information storage means is divided into long-term motion information indicating proper motion for a long-term state and short-term motion information indicating proper motion for a short-term state. A non-contact state detection device. In the non-contact state detection device according to claim 1 or 2,
temperature information acquiring means for acquiring temperature information of the subject imaged by a thermal sensor;
a body temperature region extracting means for extracting a body temperature region of the subject based on the temperature information;
The non-contact state detection device, wherein the measurement region specifying means specifies the measurement region by superimposing the body temperature region extracted by the body temperature region extraction means on the moving image information. measurement area identifying means for identifying a measurement area for measuring biometric information from video information captured by a subject;
detection means for detecting a pulse wave of the subject based on a change in luminance value between frames of the moving image information in the skin area of the measurement area;
A non-contact state detection program that causes a computer to function as state estimation means for estimating the subject's state based on the detection result of the detection means.

Images