JP7281733B2 - MOBILE SYSTEM, MOBILE, MONITORING METHOD AND PROGRAM - Google Patents

Description

TECHNICAL FIELD The present disclosure generally relates to a monitoring system, mobile object, monitoring method and program, and more particularly to a monitoring system, mobile object, monitoring method and program for monitoring the state of consciousness of a subject.

Patent Literature 1 describes a monitoring system (driver monitoring device) that monitors whether a driver is looking aside or dozing off. In this monitoring system, a luminescent spot pattern is projected onto the driver's face, and an imaging means captures an image of the luminescent spot pattern projected onto the driver's face.

Further, the data processing means processes the captured image to sample the driver's face, obtains three-dimensional position data of each sampling point, and processes the three-dimensional position data to obtain the top and bottom of the driver's face. , left, right, and oblique tilts. Then, the judging means judges the state of the driver based on the inclination of the face.

JP-A-10-960

In the monitoring system as described in Patent Literature 1, for example, even if the target person (driver) is temporarily looking down, it is erroneously determined that the target person is dozing off. There is

The present disclosure has been made in view of the above reasons, and aims to provide a monitoring system, a moving object, a monitoring method, and a program that make it difficult for erroneous determinations to occur.

A monitoring system according to one aspect of the present disclosure includes a face information acquisition unit, a model acquisition unit, a determination unit, and a face position acquisition unit . The face information acquisition unit acquires face information. The face information is information about the state of each part of the subject's face. The model acquisition unit acquires the model information. The model information is information relating to a posture model that includes a plurality of feature points and represents at least the posture of the subject's upper body. The determination unit determines the state of consciousness of the subject based on both the face information and the model information. The face position acquisition unit acquires face position information regarding at least one of the position and orientation of the face of the subject. The face position information is used for determination of the state of consciousness by the determination unit. The determination unit changes a determination algorithm based on a correlation between the face position included in the face position information and the face position included in the model information.

A mobile object according to an aspect of the present disclosure includes the monitoring system and a mobile body on which the monitoring system is mounted.

A monitoring method according to an aspect of the present disclosure includes first acquisition processing, second acquisition processing, determination processing, and face position acquisition processing . The first acquisition process is a process of acquiring face information. The face information is information about the state of each part of the subject's face. The second acquisition process is a process of acquiring model information. The model information is information relating to a posture model that includes a plurality of feature points and represents at least the posture of the subject's upper body. The determination process is a process of determining the subject's state of consciousness based on both the face information and the model information. The face position acquisition processing is processing for acquiring face position information regarding at least one of the position and orientation of the face of the subject. The face position information is used to determine the state of consciousness in the determination process. In the determination process, a determination algorithm is changed based on the correlation between the face position included in the face position information and the face position included in the model information.

A program according to an aspect of the present disclosure is a program for causing one or more processors to execute the monitoring method.

According to the present disclosure, there is an advantage that it is possible to make erroneous determinations less likely to occur.

1A is a schematic block diagram showing the configuration of a monitoring system according to Embodiment 1. FIG. 1B is a schematic perspective view showing the appearance of the moving body according to the first embodiment; FIG. FIG. 2 is an explanatory diagram showing an image used in the same surveillance system. FIG. 3A is an explanatory diagram of an enlarged eye area of a subject in an image used in the above monitoring system. FIG. 3B is an explanatory diagram in which a posture model is superimposed on an image used in the monitoring system; FIG. 3C is an explanatory diagram of an enlarged subject's face in the image used in the monitoring system. FIG. 4 is a flow chart showing an example of the overall operation of the above monitoring system. FIG. 5 is a flow chart showing an example of the operation of determining the subject's state of consciousness by the above monitoring system. FIG. 6A is an explanatory diagram for explaining an operation example of the monitoring system when the subject's state of consciousness is normal. FIG. 6B is an explanatory diagram for explaining an operation example of the monitoring system when the subject's consciousness state is loss of consciousness. FIG. 7A is an explanatory diagram for explaining an operation example of the monitoring system when the subject's state of consciousness is normal. FIG. 7B is an explanatory diagram for explaining an operation example of the monitoring system when the target person's state of consciousness is loss of consciousness. FIG. 8 is an explanatory diagram for explaining the complementary function of the monitoring system; FIG. 9 is a flowchart showing an example of the operation of determining the subject's state of consciousness by the monitoring system according to the second embodiment.

(Embodiment 1)
(1) Overview First, an overview of a monitoring system 1 according to this embodiment will be described with reference to FIGS. 1A to 3C.

The monitoring system 1 is a system for monitoring the state of consciousness of a subject H1 (see FIG. 2). Specifically, according to the monitoring system 1, it is possible to monitor whether or not the subject H1 is conscious (loss of consciousness/clear consciousness).

For example, in the field of automobiles, etc., this type of monitoring system 1 is beginning to spread as a driver monitoring system. The driver monitoring system monitors the state (including the state of consciousness) of the driver (driver), who is the target person H1, to detect whether the driver is looking aside or falling asleep. warns and controls the vehicle. As an example, the monitoring system 1 according to this embodiment can be used as such a driver monitoring system by being mounted on an automobile.

A monitoring system 1 according to the present embodiment includes a face information acquisition unit 13, a model acquisition unit 15, and a determination unit 16, as shown in FIG. 1A. The face information acquisition unit 13 acquires face information. The face information is information about the state of each part of the face F1 (see FIG. 2) of the subject H1. The model acquisition unit 15 acquires model information. The model information is information about a posture model M1 (see FIG. 3B) that includes a plurality of feature points P1 to P7 (see FIG. 3B) and represents at least the posture of the upper body of the subject H1. The determination unit 16 determines the state of consciousness of the subject H1 based on both the face information and the model information.

According to this configuration, the consciousness of the subject H1 is determined based on not only the face information about the state of each part of the face F1 of the subject H1, but also the model information about the posture model M1 representing at least the posture of the upper body of the subject H1. status is determined. Therefore, compared to the case where the state of consciousness of the subject H1 is determined only from the face information, it becomes easier to correctly determine the state of consciousness of the subject H1. Therefore, for example, when the subject H1 is simply looking down temporarily, it is difficult to make an erroneous determination that the subject H1 is dozing off. As a result, according to the monitoring system 1, it is possible to make erroneous determinations less likely to occur.

(2) Details Details of the monitoring system 1 and the moving object 4 according to the present embodiment will be described below with reference to FIGS. 1A to 7B.

(2.1) Assumptions As shown in FIG. 1B, the monitoring system 1 according to the present embodiment is mounted on a mobile body 41 and constitutes the mobile body 4 together with the mobile body 41 . That is, the mobile body 4 according to this embodiment includes the monitoring system 1 and a mobile body main body 41 . A monitoring system 1 is mounted on the mobile body 41 . In this embodiment, as an example, the mobile body 4 is an aircraft, and the mobile body main body 41 is the airframe of the aircraft. Here, in particular, it is assumed that the moving body 4 is a commercial airliner (airplane) for transporting passengers.

A subject H1 whose state of consciousness is to be monitored by the monitoring system 1 according to the present embodiment is a pilot of a moving body 4 made of an aircraft. The operator of the mobile object 4, which is the target person H1, gets into the cockpit of the mobile object 4 and controls the mobile object 4, which is an aircraft. In this embodiment, as an example, two operators, a captain and a copilot, are on board the mobile object 4, and the monitoring system 1 sets each of the two operators as a target person H1, and Suppose we want to monitor the state of consciousness of H1. However, hereinafter, unless otherwise specified, the description will focus on one target person H1 (here, the operator).

The "state of consciousness" in the present disclosure means, for example, the level of consciousness (level of consciousness) or the degree of arousal of the subject H1, and is roughly divided into a state of loss of consciousness (including loss of consciousness or fainting) / state of clear consciousness. be. The level of consciousness referred to here includes, for example, alert, somnolence, stupor, coma, and the like. Furthermore, lucid consciousness is a state of being conscious, first of all, being “awakened”, and being able to “recognize” the surroundings, and being able to respond to stimuli or information from the outside world by opening the eyes, words and actions, etc. It is also necessary to be able to "react". Conversely, unconsciousness refers to the state of being unconscious in some way. In the present disclosure, sleep (including dozing off), which is a phenomenon in which the level of consciousness is temporarily lowered and is always awakeable, is also included in unconsciousness. In this embodiment, as an example, the monitoring system 1 determines whether the subject H1's state of consciousness is in one of two states: a "normal" state corresponding to clear consciousness and a "loss of consciousness" state. .

The “state of each part of the face F1” in the present disclosure refers to the state of each part of the face F1 of the subject H1, such as the eyelids, mouth, eyeballs (eyes), nose, eyebrows, between the eyebrows, or ears. state. For example, in the state of "eyelids", the distance D1 between the upper eyelids F11 and the lower eyelids F12 (see FIG. 3A), the shape of the eyelids, and the "blinking" estimated from the distance D1 between the eyelids including the presence/absence of eyes and the distinction between eyes closed/eyes open. Mouth conditions include the position of the corners of the mouth, the shape of the lips, whether the mouth is closed or open, whether the mouth is foaming, and whether blood is being vomited. In the case of the "eyeball" state, the movement of the eyeball (movement of the line of sight), whether or not the white of the eye is turned, the degree of dilation of the pupil, and the like are included. As an example in this embodiment, the state of each part of the face F1 represented by the face information includes at least the state of the "eyelids" of the subject H1.

The “posture model” referred to in the present disclosure consists of model data representing at least the posture of the subject H1's upper body, and includes at least a plurality of feature points P1 to P7. In this embodiment, as an example, the posture model M1 is a skeletal line model in which a plurality of feature points P1 to P7 are connected by straight lines.

A “feature point” in the present disclosure is a point corresponding to each part of the human body, and is associated with, for example, the center of the face F1, the center of a pair of left and right clavicles, shoulders or elbows. In this embodiment, as an example, the posture model M1 includes seven feature points P1 corresponding to seven parts set on the upper body (from the navel to the top of the head) of the subject H1, as shown in FIG. 3B. ~P7 included. The feature point P1 is the center of the face F1, the feature point P2 is the center of a pair of left and right clavicles, the feature point P3 is the right shoulder, the feature point P4 is the right elbow, the feature point P5 is the left shoulder, the feature point P6 is the left elbow, and the feature point P7. corresponds to navel, respectively. Of these seven feature points P1 to P7, the posture model M1 is configured.

As an example in the present embodiment, such model information about the posture model M1 is extracted from an image Im1 (see FIG. 2) captured by the imaging device 2 (see FIG. 1A). The “image” referred to in the present disclosure includes moving images (moving images) and still images (still images). Furthermore, "moving image" includes an image composed of a plurality of still images obtained by frame-by-frame shooting or the like. In this embodiment, as an example, the image Im1 is an image (that is, moving image) that changes over time.

(2.2) Configuration Next, configurations of the monitoring system 1 and the moving body 4 according to this embodiment will be described.

An imaging device 2 and a control system 3 are connected to the monitoring system 1, as shown in FIG. 1A. The imaging device 2 and the control system 3 are mounted on a mobile body 41 of the mobile body 4 together with the monitoring system 1 . That is, the moving body 4 according to this embodiment further includes the imaging device 2 and the control system 3 in addition to the monitoring system 1 and the moving body main body 41 .

In this embodiment, the imaging device 2 and the control system 3 are not included in the components of the monitoring system 1, but at least one of the imaging device 2 and the control system 3 is included in the components of the monitoring system 1. may be

In this embodiment, both face information and model information are extracted from the image Im1. That is, face information and model information are used to determine the state of consciousness of the subject H1 in the monitoring system 1, and both of these face information and model information are extracted from the image Im1 captured by the imaging device 2. be done. In other words, face information used for determination of the state of consciousness by the determination unit 16 is extracted from the image Im1, and model information used for determination of the state of consciousness by the determination unit 16 is also extracted from the image Im1.

Furthermore, in the present embodiment, the information used by the determination unit 16 for determination is extracted only from the image Im1. That is, as described above, both the face information and the model information used for the determination of the state of consciousness by the determination unit 16 are information extracted from the image Im1. Furthermore, in the monitoring system 1 according to the present embodiment, in addition to face information and model information, face position information is used for determination of the state of consciousness by the determination unit 16, although details will be described later. This face position information is also extracted from the image Im1 in the same manner as the face information and the model information. Therefore, in the monitoring system 1, for example, the subject can be It is possible to determine the state of consciousness of the person H1.

The imaging device 2 is a device for capturing an image Im1 for extracting model information and the like, and includes a camera. In this embodiment, the imaging device 2 is installed so that at least the upper half of the body (including the head) of the subject H1 is included within the angle of view. In this embodiment, the target person H1 is the operator of the mobile object 4 (aircraft), so the camera of the imaging device 2 is arranged so that the target person H1 (operator) can be photographed from the front in the cockpit of the mobile object 4. , is installed in front of the subject H1. Furthermore, since each of the two operators is the subject H1, it is preferable to install one camera per subject H1.

In this embodiment, the imaging device 2 includes, for example, a night vision camera such as an infrared camera so as not to interfere with the operation of the subject H1 (operator) and to allow imaging even in a dark place. More specifically, as an example, the image Im1 captured by the imaging device 2 is a 30 fps moving image composed of a plurality of time-series still images such as Motion JPEG (Motion JPEG). An image Im1 captured by the imaging device 2 is output to the monitoring system 1, for example, in real time or via a buffer or the like.

The control system 3 is a system that operates upon receiving an output signal from the monitoring system 1 . The output signal from the monitoring system 1 contains at least the determination result of the subject H1's state of consciousness (normal/unconscious). Therefore, the control system 3 can operate according to the determination result of the consciousness state of the subject H1.

In the present embodiment, the control system 3 notifies appropriately according to the determination result of the consciousness state of the subject H1. As an example, if the determination result of the state of consciousness of the target person H1 who is the operator is "loss of consciousness", the control system 3 determines whether other crew members (other pilots, cabin crew members, etc.) and the control tower A notification is sent to one of them to inform them of the abnormality. Furthermore, the control system 3 may control equipment systems (such as a display system) other than the monitoring system 1 mounted on the mobile body 4 according to the determination result of the consciousness state of the subject H1. The moving body main body 41 may be controlled such as by switching the operation of the main body 41 to autopilot.

In this embodiment, as shown in FIG. 1A, the monitoring system 1 includes an input unit 11, an extraction unit 12, a face position acquisition unit 14 and an output A portion 17 is further provided. That is, the monitoring system 1 according to this embodiment includes an input unit 11, an extraction unit 12, a face information acquisition unit 13, a face position acquisition unit 14, a model acquisition unit 15, a determination unit 16, and an output unit 17. and have.

In this embodiment, as an example, the monitoring system 1 is mainly composed of a computer system (including servers and cloud computing) having one or more processors and one or more memories. The processor implements the functions of the monitoring system 1 by executing programs recorded in the memory. The program may be recorded in memory in advance, recorded in a non-temporary recording medium such as a memory card and provided, or provided through an electric communication line. In other words, the program is a program for causing one or more processors to function as the monitoring system 1 .

The input unit 11 is connected to the imaging device 2 . An image Im<b>1 captured by the imaging device 2 is input to the input unit 11 . In the present embodiment, the input unit 11 acquires an image Im1 composed of a moving image (moving image) from the imaging device 2 by communication in real time. Here, the image Im1 input to the input unit 11 may not be the data output from the imaging device 2 itself. For example, the image Im1 input to the input unit 11 can be appropriately compressed, converted to another data format, processed by cutting out a part of the captured image, adjusted in focus, adjusted in brightness, or adjusted in contrast as necessary. Processing such as adjustment may be applied.

The extraction unit 12 extracts, from the image Im<b>1 input to the input unit 11 , various information necessary for determining the state of consciousness of the subject H<b>1 . The extraction unit 12 extracts these various types of information by appropriately performing image processing on the image Im1. In this embodiment, as described above, face information, model information, and face position information are used for determination of the state of consciousness by the determination unit 16 . Therefore, the extraction unit 12 extracts the face information, the model information, and the face position information from the image Im1. The face information is information about the state of each part of the face F1 of the subject H1. The model information is information about the posture model M1 representing at least the posture of the subject H1's upper body. The face position information is information regarding at least one of the position and orientation of the face F1 of the subject H1. That is, for example, as shown in FIG. 2, the extraction unit 12 sets a face region R1 including the face F1 of the subject H1 in the image Im1, and extracts at least face information and face position information from the face region R1. Extract.

The extraction unit 12 extracts various kinds of information necessary for determining the state of consciousness of the subject H1 using, for example, a HOG (Histograms of Oriented Gradients) feature amount, a convolutional neural network (CNN), or the like. may

The face information acquisition unit 13 acquires face information regarding the state of each part of the face F1 of the subject H1. The face information is used by the determination unit 16 to determine the state of consciousness. In the present embodiment, the face information is extracted from the image Im1 by the extractor 12 , so the face information acquirer 13 acquires the face information extracted by the extractor 12 from the extractor 12 .

As described above, in the present embodiment, as an example, the state of each part of the face F1 represented by the face information includes at least the state of the "eyelids" of the subject H1. That is, the face information includes information regarding the condition of the eyelids of the subject H1. Therefore, the face information acquired by the face information acquisition unit 13 includes an inter-eyelid distance D1, which is the distance between the upper eyelid F11 and the lower eyelid F12, as shown in FIG. 3A. As an example, when the inter-eyelid distance D1 is equal to or less than a predetermined distance for a certain period of time, it is estimated that the subject H1 is in the "eyes closed" state.

The face position acquisition unit 14 acquires face position information regarding at least one of the position and orientation of the face F1 of the subject H1. The face position information is used by the determination unit 16 to determine the state of consciousness. In this embodiment, the face position information is extracted from the image Im1 by the extraction unit 12, so the face position acquisition unit 14 acquires the face position information extracted by the extraction unit 12 from the extraction unit 12. FIG. Here, as an example in this embodiment, as shown in FIG. 3C, the face position information is information represented by a vector A1 extending from the center of the face F1 of the subject H1 in the direction in which the face F1 is directed. contains. That is, the face position information includes both information on the position of the center of the face F1 of the subject H1 and information on the orientation of the face F1. In the example of FIG. 3C, the face F1 of the subject H1 faces the camera of the imaging device 2, so the vector A1 included in the face position information is an arrow extending straight from the center of the face F1.

The model acquisition unit 15 acquires model information about a posture model M1 that includes a plurality of feature points P1 to P7 and represents at least the posture of the upper body of the subject H1. The model information is used for determination of the state of consciousness by the determination unit 16 . In the present embodiment, the model information is extracted from the image Im1 by the extraction unit 12 , so the model acquisition unit 15 acquires the model information extracted by the extraction unit 12 from the extraction unit 12 . As described above, in this embodiment, as an example, the posture model M1 includes seven feature points P1 to P7 respectively corresponding to seven parts set on the upper body of the subject H1. Therefore, the model information acquired by the model acquisition unit 15 is, as shown in FIG. 3B, information relating to the posture model M1 generated superimposed on the upper body of the subject H1 in the image Im1.

Here, the angles θ1 and θ2 (see FIG. 3B) of the straight line connecting the feature points P3 and P4 and the straight line connecting the feature points P5 and P6 in the posture model M1 with respect to the vertical line are the arms (upper arms) of the subject H1. ). Furthermore, the angle θ3 (see FIGS. 6B and 7B) of the straight line connecting the feature points P2-P7 in the posture model M1 with respect to the vertical line corresponds to the inclination angle of the upper body of the subject H1. In the present embodiment, the model information acquired by the model acquisition unit 15 also includes information on the angles θ1 and θ2 of the arms (upper arms) of the subject H1 and information on the angle θ3 of the upper body of the subject H1. .

Here, in the present embodiment, the extraction unit 12 extracts the face information, the model information, and the face position information used for the determination of the state of consciousness by the determination unit 16 from the same image Im1. That is, since face information, model information, and face position information can be extracted from one image Im1, there is no need to install a plurality of cameras to extract each of the face information, model information, and face position information. . Therefore, in this embodiment, at least the face information acquisition unit 13 and the model acquisition unit 15 acquire face information and model information from the same image Im1.

3A to 3C and the like show a specific example of the image Im1, but the face region R1, posture model M1, vector A1, and reference numerals are only shown for the sake of explanation. It is not included in the image Im1.

The determination unit 16 has a function of determining the state of consciousness of the subject H1. The determination unit 16 determines the state of consciousness of the subject H1 based on at least both the face information and the model information. In this embodiment, the determination unit 16 basically determines the state of consciousness of the subject H1 based on face information, model information, and face position information. In the present embodiment, as an example, the determination unit 16 determines which of two states, a "normal" state corresponding to clear consciousness and a "loss of consciousness" state, is the state of consciousness of the subject H1. .

Here, the determination unit 16 accumulates each of the face information, the model information, and the face position information for a predetermined determination time, integrates the accumulated information, and determines (estimates) the state of consciousness of the subject H1. do. As an example, if the determination time is 5 minutes, the determination unit 16 only accumulates face information, model information, and face position information during the determination time of 5 minutes, and does not perform determination. Then, when 5 minutes corresponding to the determination time have passed, the accumulated face information, model information, and face position information are integrated to determine (estimate) the state of consciousness of the subject H1. Therefore, for example, even if a situation that can be determined as loss of consciousness occurs temporarily within the determination time, if the determination is normal (clear consciousness) throughout the determination time, the determination unit 16 is normal. (clear consciousness) can be determined. On the other hand, even if a situation where it can be determined that the consciousness is clear temporarily occurs during the determination time, the determination unit 16 determines that the consciousness is lost if the loss of consciousness is determined throughout the determination time. be able to.

Further, details will be described in the section "(2.3.2) Judgment operation", but the judgment unit 16 considers the reliability of each of the face information, the model information, and the face position information, It is possible to change the priority or weighting of face information, model information, and face position information when determining the state of consciousness. That is, in the present embodiment, the determination unit 16 does not always use all of the face information, the model information, and the face position information when determining the state of consciousness of the subject H1. change the priority or weighting of As a result, the determination unit 16 may determine the state of consciousness of the subject H1 based only on the face information among the face information, the model information, and the face position information, for example.

The determination unit 16 preferably has a function of learning by machine learning such as deep learning. In this case, the determining unit 16 determines the state of consciousness of the subject H1 by inputting the face information, the model information, and the face position information into the learned classifier.

The output unit 17 receives the determination result of the determination unit 16 and outputs an output signal to the control system 3 . The output unit 17 includes at least the determination result (normal/unconsciousness) of the state of consciousness of the subject H1 in the output signal, and transmits the output signal to the control system 3 by communication. Preferably, the output unit 17 immediately outputs an output signal to the control system 3 at least when the determination result is loss of consciousness. On the other hand, when the determination result is normal (clear consciousness), the output unit 17 does not have to output the output signal to the control system 3 immediately.

Furthermore, the output unit 17 may output the probability (probability) of the determination result in addition to the determination result of the determination unit 16 indicating the state of consciousness (normal/unconscious) of the subject H1. For example, when the determination unit 16 determines that “loss of consciousness” has a probability of 70%, the output unit 17 outputs the determination result of “loss of consciousness” and adds that the probability of this determination result is “70%”. to output

(2.3) Operation Next, the operation of the monitoring system 1 according to this embodiment, that is, the monitoring method according to this embodiment will be described.

In the following, the monitoring period is defined as a period from when the moving object 4, which is an aircraft, takes off until it starts to land. Assuming you do. In particular, it is assumed here that the mobile body 41 is cruising by autopilot during the period from when the mobile body 4 takes off until it starts landing. In other words, the monitoring system 1 monitors the state of consciousness of the subject H1 who is the operator of the moving body 4 during the monitoring period when the moving body 4 is cruising by autopilot.

Furthermore, it is assumed here that the determination time, which is a period in which the determination unit 16 determines the state of consciousness of the subject H1, is 5 minutes. That is, the determination unit 16 accumulates each of the face information, the model information, and the face position information for the determination time of 5 minutes, and integrates the accumulated information each time the determination time of 5 minutes elapses. Determine the state of consciousness of the subject H1.

(2.3.1) Overall Operation FIG. 4 is a flowchart showing an example of the overall operation of the monitoring system 1 during the monitoring period, that is, the overall flow of the monitoring method.

That is, when the monitoring period starts, the monitoring system 1 first acquires an image Im1 captured by the imaging device 2 from the imaging device 2 by the input unit 11 (S1). When the image Im1 is acquired, the monitoring system 1 extracts face information, model information, and face position information from the acquired image Im1 by the extraction unit 12 .

Then, the monitoring system 1 acquires face information with the face information acquisition unit 13 (S2), acquires face position information with the face position acquisition unit 14 (S3), and acquires model information with the model acquisition unit 15. (S4). After that, the monitoring system 1 determines whether or not the determination time (here, 5 minutes) has passed (S5). If the determination time has not elapsed, the process returns to step S1 and repeats the processes after the acquisition of the image Im1.

On the other hand, when the determination time has elapsed, the monitoring system 1 executes determination processing for determining the state of consciousness of the subject H1 in the determination unit 16 (S6). In this determination process S6, the determination unit 16 can determine the state of consciousness of the subject H1 based on at least both the face information and the model information. The determination processing S6 will be described in detail in the section "(2.3.2) Determination operation".

When the determination process S6 is completed, the monitoring system 1 outputs an output signal including at least the determination result (normal/unconsciousness) of the state of consciousness of the subject H1 to the control system 3 through the output unit 17 (S7). As a result, for example, if the determination result in the determination process S6 is "loss of consciousness", the control system 3 is directed to at least one of the other crew members (other pilots, cabin crew members, etc.) and the control tower. , make a report to notify of an abnormality.

The monitoring system 1 repeatedly executes the processes S1 to S7 as described above over the monitoring period. The flowchart of FIG. 4 is merely an example of the overall operation of the monitoring system 1, and processing may be omitted or added as appropriate, and the order of processing may be changed as appropriate. For example, the order of the processes S2 to S4 is changed, and after model information is acquired by the model acquisition unit 15 (S4), face position information is acquired by the face position acquisition unit 14 (S3), and then face information is acquired. Face information may be acquired by the unit 13 (S2).

(2.3.2) Determining Operation Hereinafter, the operation of determining the state of consciousness of the subject H1 by the monitoring system 1, that is, the determining process S6 in FIG. 4 will be described in detail.

FIG. 5 is a flow chart showing an example of the operation of the determination unit 16 in the determination process (S6 in FIG. 4). In the flowchart of FIG. 5, the processes (S16, S17) in which the determination unit 16 uses the model information to determine the state of consciousness of the subject H1 are shaded. In other words, in FIG. 5, the non-shaded processes are processes in which the model information is not used to determine the state of consciousness of the subject H1.

When the determination process starts, the determination unit 16 first determines whether or not the "eyelids" of the subject H1 have been detected (S11). For example, if the eyelids (upper eyelid position F11, lower eyelid position F12) can be clearly recognized from the image Im1, it is determined that the eyelids have been detected (S11: Yes), and if the eyelids cannot be clearly recognized in the image Im1, It is determined that the eyelids cannot be detected (S11: No). Whether the eyelids can be detected is included in the face information acquired by the face information acquisition unit 13, for example.

If the eyelids are detected (S11: Yes), the determination unit 16 determines whether or not the subject is blinking and whether or not the eyes are closed (S12). The presence or absence of blinking and the presence or absence of closed eyes are determined based on the face information acquired by the face information acquiring section 13 . For example, if the subject H1 does not "blink" for a predetermined time (one minute as an example), it is determined that there is no "blinking", and the distance D1 between the eyelids is equal to or less than the predetermined distance for a certain time (one minute as an example). minutes), it is determined that the eyes are closed (S12: Yes). On the other hand, if it is determined that the subject is "blinking" and that the eye is open, the determination in step S12 is "No".

Then, if it is determined that there is no "blinking" or that the eyes are closed (S12: Yes), the determination unit 16 determines that the subject H1 is in a state of unconsciousness (S18). On the other hand, when it is determined that there is "blinking" and that the eyes are open (S12: No), the determining unit 16 determines that the subject H1 is in a normal state of consciousness (clear consciousness) (S19). ).

Here, in the case of determining the state of consciousness from the determination result of whether or not blinking and whether or not the eyes are closed (S12), the determination unit 16 determines the state of consciousness of the subject H1 only from the face information. That is, the determining unit 16 can determine the state of consciousness of the subject H1 based not only on face information but also on model information and face position information. highest degree. Therefore, when the reliability of the face information is equal to or higher than the threshold, the face information is preferentially used to determine the state of consciousness of the subject H1. It is possible to determine the state. The reliability of the determination based on the face information is estimated by whether or not the "eyelids" of the subject H1 have been detected (S11). Confidence is assumed to be greater than or equal to the threshold.

On the other hand, if the eyelids are not detected (S11: No), that is, if the reliability of the determination based on the face information is less than the threshold, the determination unit 16 determines that the orientation of the face F1 of the subject H1 is stiff. It is determined whether or not there is (S13). The rigidity of the orientation of the face F1 is determined based on the face position information acquired by the face position acquisition section 14 . For example, if the integrated value of the amount of change in the orientation of the face F1 over a certain period of time (one minute, for example) is less than a predetermined value, it is determined that the orientation of the face F1 is stiff (S13: Yes), and the integrated value reaches the predetermined value. If it is equal to or greater than the value, it is determined that the orientation of the face F1 is not rigid (S13: No).

When it is determined that the orientation of the face F1 is rigid (S13: Yes), the determination unit 16 determines that the subject H1 is in a state of unconsciousness (S18). Here, when determining the state of consciousness from the determination result of the presence or absence of rigidity in the orientation of the face F1 (S13), the determination unit 16 determines the state of consciousness of the subject H1 from the face position information. That is, in this embodiment, as a default, the priority of face information is set to be the highest, followed by the priority of face position information. Therefore, when the eyelids are not detected (S11: No) and the reliability of the determination based on the face information is less than the threshold, the face position information is preferentially used to determine the state of consciousness of the subject H1. The unit 16 can determine the state of consciousness of the subject H1 based on the face position information.

On the other hand, when it is determined that the orientation of the face F1 is not rigid (S13: No), the determination unit 16 determines whether or not the position of the face F1 of the subject H1 has been detected (S14). For example, if the face F1 of the subject H1 can be clearly recognized from the image Im1, it is determined that the position of the face F1 has been detected (S14: Yes). It is determined that the position of F1 cannot be detected (S14: No). Whether or not the position of the face F1 can be detected is included in the face position information acquired by the face position acquisition unit 14, for example.

If the position of the face F1 can be detected (S14: Yes), the determining unit 16 determines whether there is a correlation between the position of the face F1 included in the face position information and the position of the face F1 included in the model information. It is determined whether or not (S15). For example, if the distance between the center position of the face F1 included in the face position information and the position of the feature point P1 included in the posture model M1 is less than a predetermined value, it is determined that there is a correlation between the two (S15: Yes), if the interval is equal to or greater than the predetermined value, it is determined that there is no correlation between the two (S15: No).

When it is determined that there is a correlation between the two face positions (S15: Yes), the determination unit 16 determines whether or not the body (upper body) of the subject H1 is rigid (S16). Body stiffness is determined based on the model information acquired by the model acquisition unit 15 . For example, the position of the plurality of feature points P1 to P7 in the posture model M1 and/or the integral value of the amount of change in the angles θ1 and θ2 of the arm (upper arm) of the subject H1 over a certain period of time (one minute as an example) is predetermined. If it is less than the value, it is determined that the body is rigid (S16: Yes). On the other hand, if the integrated value is equal to or greater than the predetermined value, it is determined that the body is not rigid (S16: No).

When it is determined that the body is rigid (S16: Yes), the determination unit 16 determines whether or not the upper body of the subject H1 is tilted (S17). The inclination of the upper body is determined based on the angle θ3 of the straight line connecting the feature points P2 to P7 in the posture model M1 with respect to the vertical line among the model information acquired by the model acquisition unit 15. FIG. For example, if the angle θ3 is less than the reference angle, it is determined that the upper body of the subject H1 is not tilted (S17: No), and if the angle θ3 is equal to or greater than the reference angle, it is determined that the upper body of the subject H1 is tilted. (S17: Yes).

When it is determined that the body of the subject H1 is stiff (S16: Yes) and that the upper body of the subject H1 is tilted (S17: Yes), the determination unit 16 determines the state of consciousness of the subject H1. is in a state of unconsciousness (S18). On the other hand, when it is determined that the subject H1 does not have body stiffness (S16: No) or that the subject H1 does not tilt the upper body (S17: No), the determination unit 16 determines that the subject H1 has a normal state of consciousness. (S19).

Here, in the case of determining the state of consciousness from the determination results of the presence or absence of body stiffness and the presence or absence of tilting of the upper body (S16, S17), the determination unit 16 determines the state of consciousness of the subject H1 from the model information. will do. That is, in this embodiment, as a default, the priority of face information is set to be the highest, followed by the priority of face position information, and the priority of model information to the lowest. Therefore, if the state of consciousness of the subject H1 cannot be determined from the face information and the face position information, and the reliability of the model information is equal to or higher than the threshold, the model information is used to determine the state of consciousness of the subject H1. is used, and the determination unit 16 determines the state of consciousness of the subject H1 from the model information. The reliability of the determination based on the model information is estimated by whether there is a correlation between the position of the face F1 included in the face position information and the position of the face F1 included in the model information (S15). If there is a correlation between the two (S15: Yes), it is estimated that the reliability of the determination based on the model information is greater than or equal to the threshold.

On the other hand, if the position of the face F1 cannot be detected (S14: No), or if there is no correlation between the face positions (S15: No), the determination unit 16 determines that the state of consciousness of the subject H1 is normal (clear consciousness). (S19). In these cases, the determination unit 16 determines the state of consciousness of the subject H1 without using model information. In other words, when the face position cannot be detected (S14: No), or when there is no correlation between the face positions (S15: No), the determination unit 16 determines that the reliability of the determination based on the model information is less than the threshold. determines the state of consciousness of the subject H1 without using model information.

The monitoring system 1 executes the processes S11 to S19 as described above in the determination process. The flowchart of FIG. 5 is merely an example of determination processing, and processing may be omitted or added as appropriate, and the order of processing may be changed as appropriate.

As described above, in this embodiment, when determining the state of consciousness of the subject H1, the determination unit 16 considers the reliability of each of the face information, the model information, and the face position information. and the priority or weighting of face position information can be changed. In the example of FIG. 5, the priorities of face information, model information, and face position information are defined by default such that the order of priority is face information, face position information, and model information. Then, when the reliability of determination based on face information is low, face position information is preferentially used instead of face information. The priority or weighting of location information is changed.

In particular, in the present embodiment, the determination unit 16 determines the state of consciousness based on the face information if the reliability of the determination based on the face information is equal to or greater than the threshold, and if the reliability of the determination based on the face information is less than the threshold. For example, the state of consciousness is determined based on the model information. In this way, while the priority of face information is higher than that of model information by default, only when the reliability of determination based on face information is insufficient, the determination unit 16 uses information other than face information (model information). ), the state of consciousness of the subject H1 is determined. As a result, determination using face information with relatively high reliability is preferentially performed, and the reliability of determination of the state of consciousness of the subject H1 in the monitoring system 1 increases.

Further, in the present embodiment, as described above, the determination unit 16 uses the determination algorithm based on the correlation between the position of the face F1 included in the face position information and the position of the face F1 included in the model information. change. That is, if there is a correlation between the two (S15: Yes), the determination unit 16 determines the state of consciousness of the subject H1 based on the model information (S16, S17). (S15: No), the model information is not used for the state of consciousness of the subject H1. In short, when there is a correlation between the two (S15: Yes), the reliability of the model information is estimated to be sufficiently high, and the determination algorithm in the determination unit 16 changes so that the model information is used.

Furthermore, the determination unit 16 determines the state of consciousness based on at least one of the degree of stiffness of the upper body of the subject H1 and the posture of the upper body of the subject H1 in the model information. In short, when determining the state of consciousness of the subject H1 based on the model information, the determining unit 16 determines the state of consciousness based on both the presence or absence of body rigidity and the presence or absence of tilting of the upper body (S16, S17). Here, the presence or absence of stiffness of the body (S16) corresponds to the degree of stiffness of the upper body of the subject H1, and the presence or absence of leaning of the upper body (S17) corresponds to the posture of the upper body of the subject H1. Thus, in this embodiment, both the degree of stiffness of the upper body of the subject H1 and the posture of the upper body of the subject H1 in the model information are used to determine the state of consciousness of the subject H1.

In addition, in the flowchart of FIG. 5, in the process S12, the determination unit 16 collectively determines the presence or absence of blinking and the presence or absence of closed eyes. You can judge At this time, the determination unit 16 may change the determination algorithm depending on whether blinking of the subject H1 is not detected or when closing the eyes of the subject H1 is detected.

As an example, even if blinking is not detected or eye closure is detected, the determination unit 16 does not immediately determine the state of unconsciousness, but information other than face information, that is, face position information or The state of consciousness of the subject H1 is determined based on the model information. The determining unit 16 preferentially uses the face position information to determine the state of consciousness of the subject H1 when blinking is not detected, and determines the state of consciousness of the subject H1 when closed eyes are detected. preferentially use model information. In this way, the determination unit 16 determines the information (face information, face position information, and model information) used for determining the state of consciousness of the subject H1 after that when blinking is not detected and when eye closure is detected. priority can be changed. In addition, the determination unit 16 determines whether information (face information, face position information, and model information) used to determine the state of consciousness of the subject H1 after that, when blinking is not detected and when eye closure is detected, Reliability or determination time may be changed.

(2.3.3) Specific Example Next, a specific example of determination of the state of consciousness of the subject H1 by the monitoring system 1 will be described with reference to FIGS. 6A to 7B.

6A and 6B show an example in which the monitoring system 1 determines the state of consciousness of the subject H1 when the subject H1 is temporarily facing downward. FIG. 6A shows a case in which the target person H1 is in a normal state of consciousness (clear consciousness), and FIG. 6B shows a case in which the target person H1 is unconscious.

In both FIGS. 6A and 6B, since the subject H1 is facing downward, the eyelids of the subject H1 are not detected (S11 in FIG. 5: No), and it is determined that the orientation of the face F1 is not stiff. (S13: No). Furthermore, in both cases, the position of the face F1 is detected (S14: Yes), and there is a correlation between the position of the face F1 included in the face position information and the position of the face F1 included in the model information (S15 : Yes), the determination unit 16 determines the state of consciousness of the subject H1 based on the model information.

Here, in the example of FIG. 6A, the determination unit 16 determines that the subject H1 It is determined that there is no stiffness in the body of the subject (S16: No). As a result, the determination unit 16 determines that the subject H1 is in a normal state (clear consciousness) (S19).

On the other hand, in the example of FIG. 6B , the determination unit 16 determines that the integrated value of the changes in the angles θ1 and θ2 of the arm (upper arm) of the subject H1 is less than the predetermined value. It is determined that the body is rigid (S16: Yes). Furthermore, since the angle θ3 of the straight line connecting the feature points P2 to P7 in the posture model M1 with respect to the vertical line is greater than or equal to the reference angle, the determination unit 16 determines that the upper body of the subject H1 is tilted (S17: Yes ). As a result, the determination unit 16 determines that the subject H1 is in a state of unconsciousness (S18).

7A and 7B show an example in which the monitoring system 1 determines the state of consciousness of the subject H1 when the subject H1 wears (wears) sunglasses. FIG. 7A shows a case in which the target person H1 is in a normal state of consciousness (clear consciousness), and FIG. 7B shows a case in which the target person H1 is unconscious.

In both FIGS. 7A and 7B, since the subject H1 is wearing sunglasses, the eyelids of the subject H1 are not detected (S11 in FIG. 5: No), and it is determined that the orientation of the face F1 is not stiff. (S13: No). Furthermore, in both cases, the position of the face F1 is detected (S14: Yes), and there is a correlation between the position of the face F1 included in the face position information and the position of the face F1 included in the model information (S15 : Yes), the determination unit 16 determines the state of consciousness of the subject H1 based on the model information.

Here, in the example of FIG. 7A, the determination unit 16 determines that the subject H1 It is determined that there is no stiffness in the body of the subject (S16: No). As a result, the determination unit 16 determines that the subject H1 is in a normal state (clear consciousness) (S19).

On the other hand, in the example of FIG. 7B , the determination unit 16 determines that the integrated value of the changes in the angles θ1 and θ2 of the arm (upper arm) of the subject H1 is less than a predetermined value over a predetermined period of time. It is determined that the body is rigid (S16: Yes). Furthermore, since the angle θ3 of the straight line connecting the feature points P2 to P7 in the posture model M1 with respect to the vertical line is greater than or equal to the reference angle, the determination unit 16 determines that the upper body of the subject H1 is tilted (S17: Yes ). As a result, the determination unit 16 determines that the subject H1 is in a state of unconsciousness (S18).

In this way, even if the target person H1 is temporarily looking down or is wearing sunglasses, the state of each part (eyelids, etc.) of the face F1 of the target person H1 cannot be recognized. However, the monitoring system 1 according to this embodiment can determine the state of consciousness of the subject H1.

By the way, regarding the angles θ1 and θ2 of the arm (upper arm) of the subject H1, the determination unit 16 not only determines whether or not the arm moves, such as the integrated value of the variation, but also determines whether the arm motion is passive. It may be further determined whether or not the movement is significant. That is, assuming that the target person H1 is on the moving body 4, the movement (vibration) of the moving body main body 41 causes the arm of the target person H1 to move like a pendulum regardless of the intention of the target person H1. It is conceivable that the Therefore, as shown in FIG. 6B or FIG. 7B, even if the subject H1 is unconscious, the arm of the subject H1 who is in a weakened state may passively move. Therefore, regarding the movement of the arm, the determination unit 16 determines a passive movement, such as a case where the arm is swaying based on a state in which it hangs down vertically. (S16: Yes). A similar determination is applicable not only to the arm, but also to the head, for example.

Further, the determination of whether or not the upper body of the subject H1 is tilted (S17) is not limited to the determination based on the angle θ3, and whether or not the upper body of the subject H1 is tilted may be determined based on other parameters. . As an example, the angle of the straight line connecting the feature points P3-P5 in the posture model M1 with respect to the horizontal corresponds to the inclination angle of the shoulders of the subject H1. You may judge the presence or absence of inclination. In this case, if the inclination angle of both shoulders is less than the reference angle, it is determined that the upper body of the subject H1 is not inclined (S17: No). It is determined that the upper body is tilted (S17: Yes).

(2.3.4) Other Functions As other functions, the monitoring system 1 according to the present embodiment has a complementary function that complements the feature points P1 to P7 of the posture model M1 in the model information.

That is, as shown in FIG. 8, the entire upper body of the subject H1 is not included within the angle of view of the imaging device 2, and any of the feature points P1 to P7 of the posture model M1 is within the image Im1. It may not fit. In particular, when the chair on which the subject H1 sits is movable back and forth, moving the chair forward brings the subject H1 closer to the camera of the imaging device 2, resulting in an image Im1 as shown in FIG. Cheap. In the example of FIG. 8, the feature point P7 corresponding to the "navel" protrudes from the image Im1.

In such a case, the monitoring system 1 estimates the position of the feature point P7 from the position of the subject H1 in the image Im1, etc., and interpolates the feature point P7. According to the complementing function described above, in the monitoring system 1 according to the present embodiment, the model information includes points corresponding to parts of the body of the subject H1 outside the image Im1 for extracting the model information as the plurality of feature points P1. ~P7. As a result, for example, there is no need to widen the camera of the imaging device 2 more than necessary, and it is easy to improve the resolution of each part of the face F1 (upper eyelid F11, lower eyelid F12, etc.) in the image Im1.

(3) Modifications Embodiment 1 is merely one of various embodiments of the present disclosure. Embodiment 1 can be modified in various ways according to design and the like, as long as the object of the present disclosure can be achieved. Each drawing described in this disclosure is a schematic drawing, and the ratio of the size and thickness of each component in each drawing does not necessarily reflect the actual dimensional ratio. Also, the same functions as those of the monitoring system 1 according to the first embodiment may be embodied by a monitoring method, a program, or a non-temporary recording medium recording the program. A monitoring method according to one aspect includes a first acquisition process (corresponding to "S2" in FIG. 4), a second acquisition process (corresponding to "S3" in FIG. 4), a determination process (corresponding to "S6" in FIG. equivalent) and The first acquisition process is a process of acquiring face information. The face information is information about the state of each part of the face F1 of the subject H1. The second acquisition process is a process of acquiring model information including a plurality of feature points P1 to P7. The model information is information about the posture model M1 representing at least the posture of the subject H1's upper body. The determination process is a process of determining the state of consciousness of the subject H1 based on both the face information and the model information. A program according to one aspect is a program for causing one or more processors to execute the above monitoring method.

Modifications of the first embodiment are listed below. Modifications described below can be applied in combination as appropriate.

A monitoring system 1 in the present disclosure includes a computer system. A computer system is mainly composed of a processor and a memory as hardware. The functions of the monitoring system 1 in the present disclosure are realized by the processor executing a program recorded in the memory of the computer system. The program may be recorded in advance in the memory of the computer system, may be provided through an electric communication line, or may be recorded in a non-temporary recording medium such as a computer system-readable memory card, optical disk, or hard disk drive. may be provided. A processor in a computer system consists of one or more electronic circuits, including semiconductor integrated circuits (ICs) or large scale integrated circuits (LSIs). The integrated circuit such as IC or LSI referred to here is called differently depending on the degree of integration, and includes integrated circuits called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). In addition, a field-programmable gate array (FPGA) that is programmed after the LSI is manufactured, or a logic device capable of reconfiguring the bonding relationship inside the LSI or reconfiguring the circuit partitions inside the LSI may also be adopted as the processor. can be done. A plurality of electronic circuits may be integrated into one chip, or may be distributed over a plurality of chips. A plurality of chips may be integrated in one device, or may be distributed in a plurality of devices. A computer system, as used herein, includes a microcontroller having one or more processors and one or more memories. Accordingly, the microcontroller also consists of one or more electronic circuits including semiconductor integrated circuits or large scale integrated circuits.

In addition, it is not essential for the monitoring system 1 that at least part of the functions of the monitoring system 1 are integrated in one housing. may be provided. For example, the face information acquisition unit 13, the face position acquisition unit 14, and the model acquisition unit 15 of the monitoring system 1 may be provided in a separate housing from the determination unit 16. FIG. Furthermore, at least part of the functions of the monitoring system 1 may be realized by a cloud (cloud computing) or the like.

Conversely, in Embodiment 1, at least some of the multiple functions distributed over multiple devices may be integrated into one housing. For example, functions distributed between the monitoring system 1 and the imaging device 2 may be integrated in one housing.

Further, the mode of outputting the determination result of the state of consciousness of the subject H1 is not limited to the output to the control system 3 . For example, the output unit 17 transmits, displays, outputs sound (including voice), records (writes) to a non-temporary recording medium, and prints (prints out) the determination result of the state of consciousness of the subject H1 to the information terminal. ), etc., may be output.

Further, the mobile object 4 equipped with the monitoring system 1 is not limited to an airplane, and may be an aircraft such as a helicopter or an airship. Furthermore, the mobile object 4 equipped with the monitoring system 1 is not limited to an aircraft, and may be, for example, an automobile (four-wheeled vehicle), a two-wheeled vehicle, a train, an electric cart, a construction machine, a ship, or the like. For example, when the mobile object 4 is an automobile, the monitoring system 1 may monitor the consciousness state of the target person H1 who is the driver of the mobile object 4, which is an automobile.

Further, the subject H1 whose state of consciousness is monitored by the monitoring system 1 mounted on the mobile body 41 is not limited to the operator (including the driver in the case of an automobile) who operates the mobile body 4. For example, , a cabin attendant or a conductor, etc., other than the pilot. Furthermore, the target person H1 is not limited to a crew member, and may be, for example, a passenger or fellow passenger.

Moreover, the monitoring system 1 is not limited to the configuration used for the mobile body 4, and may be used for monitoring the state of consciousness of the subject H1 who is in a building, outdoors, or the like, for example. In this case, the target person H1 may be a worker who is performing some work such as machine operation, experiment, test, measurement, or work, or may be a resident in a building such as a house.

In addition, in the first embodiment, the monitoring system 1 determines whether the state of consciousness of the subject H1 is "normal" (clear consciousness) or "loss of consciousness". , but not limited to this configuration. For example, the monitoring system 1 may determine three or more levels of consciousness, such as lucidity, somnolence, stupor, and coma, as the state of consciousness of the subject H1.

Further, the imaging device 2 for obtaining the image Im1 is not limited to a configuration for capturing an image of the object itself. may Further, for example, like a ToF (Time-of-Flight) type range image camera, a range image in which the distance is a pixel value may be captured as an image.

Further, in Embodiment 1, the imaging device 2 includes, for example, a night vision camera such as an infrared camera, but is not limited to this, and may include, for example, an illumination device and a camera.

Further, in the first embodiment, one camera is installed for one target person H1 in order to photograph a plurality of target persons H1 (operators). A single camera may be used to photograph a plurality of subjects H1.

Further, the information used for determination by the determination unit 16 is not limited to the configuration extracted only from the image Im1. The information may be used to determine the state of consciousness of subject H1. Furthermore, the determination unit 16 also considers the operation state of the operation unit that can be operated by the subject H1, such as the control stick, pedals, or instrument switches of the moving body 4 made of an aircraft, to determine the state of consciousness of the subject H1. You can judge. As a result, it is possible to reduce erroneous determination of unconsciousness by the monitoring system 1 when the target person H1 is operating the control stick in a substantially stationary posture.

Further, in the first embodiment, when determining the state of consciousness of the subject H1, the determination unit 16 follows the priority of each of the face information and the model information, and only when the determination based on the face information is not preferable, the model information. However, it is not limited to such a configuration. That is, for example, the determination unit 16 may determine the state of consciousness of the subject H1 based on both the face information and the model information. may be used in an integrated manner to determine the state of consciousness of the subject H1. In this case, by weighting each of the face information and the model information, the determination unit 16 may adjust the relative weight of the face information and the model information that contribute to the determination of the state of consciousness of the subject H1.

Furthermore, the determination unit 16 only needs to have at least a function of determining the state of consciousness of the subject H1 based on both the face information and the model information. good. As an example, by switching the determination algorithm, the determination unit 16 may operate to determine the state of consciousness of the subject H1 based on only one of the face information and the model information.

(Embodiment 2)
In the monitoring system 1 according to the present embodiment, as shown in FIG. 9, the operation of determining the state of consciousness of the subject H1 by the monitoring system 1, that is, the algorithm of the determination processing S6 in FIG. different from 1. In the following, configurations similar to those of the first embodiment are denoted by common reference numerals, and descriptions thereof are omitted as appropriate.

FIG. 9 is a flow chart showing an example of the operation of the determination unit 16 in the determination process (S6 in FIG. 4). In the flowchart of FIG. 9, the processes (S21, S22) in which the determination unit 16 uses the model information to determine the state of consciousness of the subject H1 are hatched. In other words, in FIG. 9, the non-shaded processes are processes in which the model information is not used to determine the state of consciousness of the subject H1.

When the determination process starts, the determination unit 16 first determines whether or not the body (upper body) of the subject H1 is stiff (S21). Body stiffness is determined based on the model information acquired by the model acquisition unit 15 (corresponding to processing S16 in FIG. 5). Regarding the presence or absence of stiffness of the body of the subject H1, as in the first embodiment, for example, regarding the movement of the arm, whether it is a passive movement such as a swinging based on a vertically hanging state. It may be determined by considering whether or not

When it is determined that the body is rigid (S21: Yes), the determination unit 16 determines whether or not the upper half of the body of the subject H1 is tilted (S22). The inclination of the upper body is determined based on the angle θ3 of the straight line connecting the feature points P2 to P7 in the posture model M1 with respect to the vertical line among the model information acquired by the model acquisition unit 15 (process S17 in FIG. 5). equivalent to ). As in the first embodiment, whether or not the upper body of the subject H1 is tilted may be determined, for example, based on the angle of the straight line connecting the feature points P3-P5 in the posture model M1 with respect to the horizontal line.

When it is determined that the body of the subject H1 is stiff (S21: Yes) and that the upper body of the subject H1 is tilted (S22: Yes), the determination unit 16 determines the state of consciousness of the subject H1. is in a state of unconsciousness (S25). On the other hand, when it is determined that the subject H1 is not stiff (S21: No), the determining unit 16 determines that the subject H1 is in a normal state of consciousness (clear consciousness) (S26).

Further, when it is determined that the upper body of the subject H1 is not tilted (S22: No), the determination unit 16 determines whether the orientation of the face F1 of the subject H1 is rigid (S23). The rigidity of the orientation of the face F1 is determined based on the face position information acquired by the face position acquisition unit 14 (corresponding to processing S13 in FIG. 5).

When it is determined that the orientation of the face F1 is rigid (S23: Yes), the determination unit 16 determines that the subject H1 is in a state of unconsciousness (S25). Here, when determining the state of consciousness from the determination result of the presence or absence of rigidity in the orientation of the face F1 (S23), the determination unit 16 determines the state of consciousness of the subject H1 from the face position information.

On the other hand, when it is determined that the orientation of the face F1 is not rigid (S23: No), the determining unit 16 determines whether or not the face is blinking and whether or not the eyes are closed (S24). The presence or absence of blinking and the presence or absence of closed eyes are determined based on the face information acquired by the face information acquisition unit 13 (corresponding to processing S12 in FIG. 5).

Then, if it is determined that there is no "blinking" or that the eyes are closed (S24: Yes), the determination unit 16 determines that the subject H1 is in a state of unconsciousness (S25). On the other hand, when it is determined that there is "blinking" and that the eyes are open (S24: No), the determination unit 16 determines that the subject H1 is in a normal state of consciousness (clear consciousness) (S26). ).

Thus, in this embodiment, as a default, model information is set to have the highest priority, face position information has the next highest priority, and face information has the lowest priority. Therefore, if the state of consciousness of the subject H1 cannot be determined from the model information and the face position information, and the reliability of the face information is equal to or higher than the threshold, the state of consciousness of the subject H1 is determined using the face information. is used, and the determination unit 16 determines the state of consciousness of the subject H1 from the face information.

The monitoring system 1 according to this embodiment executes the processes S21 to S26 as described above in the determination process. The flowchart of FIG. 9 is merely an example of determination processing, and processing may be omitted or added as appropriate, and the order of processing may be changed as appropriate.

Various configurations (including modifications) described in the second embodiment can be employed in appropriate combination with various configurations (including modifications) described in the first embodiment.

(summary)
As described above, the monitoring system (1) according to the first aspect includes a face information acquisition section (13), a model acquisition section (15), and a determination section (16). A face information acquisition unit (13) acquires face information. The face information is information about the state of each part of the face (F1) of the subject (H1). A model acquisition unit (15) acquires model information. The model information is information about a posture model (M1) that includes a plurality of feature points (P1 to P7) and represents at least the posture of the subject's (H1) upper body. A determination unit (16) determines the state of consciousness of the subject (H1) based on both the face information and the model information.

According to this aspect, not only the face information about the state of each part of the face (F1) of the subject (H1), but also the model information about the posture model (M1) representing at least the posture of the upper body of the subject (H1). Based on this, the state of consciousness of the subject (H1) is determined. Therefore, compared to the case where the state of consciousness of the subject (H1) is determined only from the face information, it becomes easier to correctly determine the state of consciousness of the subject (H1). Therefore, for example, when the target person (H1) is simply looking down temporarily, it is difficult to make an erroneous determination that the target person (H1) is dozing off. As a result, according to the monitoring system (1), it is possible to make it difficult to make an erroneous determination.

In the monitoring system (1) according to the second aspect, in the first aspect, the determination unit (16) determines the state of consciousness based on the face information if the reliability of the determination based on the face information is equal to or greater than a threshold. , the state of consciousness is determined based on the model information if the reliability of the determination based on the face information is less than the threshold.

According to this aspect, face information is preferentially used to determine the state of consciousness rather than model information, so it is easy to improve the reliability of determination of the state of consciousness.

A monitoring system (1) according to a third aspect, in the first or second aspect, further comprises a face position acquisition section (14). A face position acquisition unit (14) acquires face position information regarding at least one of the position and orientation of the face (F1) of the subject (H1). The face position information is used for determination of the state of consciousness in the determining section (16).

According to this aspect, since the face position information is further used to determine the state of consciousness, it is easy to improve the reliability of determination of the state of consciousness.

In the monitoring system (1) according to the fourth aspect, in the third aspect, the determination unit (16) determines the position of the face (F1) included in the face position information and the position of the face (F1) included in the model information. The determination algorithm is changed based on the correlation between the position and the position.

According to this aspect, since the determination algorithm can be changed according to the reliability of the model information, it is easy to improve the reliability of determination of the state of consciousness.

In the monitoring system (1) according to the fifth aspect, in any one of the first to fourth aspects, the face information includes information regarding the condition of the eyelids of the subject (H1).

According to this aspect, it is possible to estimate, for example, the presence or absence of blinking, closed eyes, or the like from the state of the eyelids of the subject (H1), and it is easy to improve the reliability of determination of the state of consciousness.

In the monitoring system (1) according to the sixth aspect, in the fifth aspect, the determination unit (16) determines whether blinking of the subject (H1) is not detected or when closed eyes of the subject (H1) are detected. Change the judgment algorithm depending on the situation.

According to this aspect, since the determination algorithm can be changed according to the situation, it is easy to improve the reliability of determination of the state of consciousness.

In the monitoring system (1) according to the seventh aspect, in any one of the first to sixth aspects, both the face information and the model information are extracted from the image (Im1).

According to this aspect, face information and model information can be obtained relatively easily from the image (Im1).

In the monitoring system (1) according to the eighth aspect, in the seventh aspect, the information used by the determination section (16) for determination is extracted only from the image (Im1).

According to this aspect, there is no need to use a sensor or the like other than the imaging device (2) for obtaining the image (Im1).

In the monitoring system (1) according to the ninth aspect, in the seventh or eighth aspect, the face information acquisition section (13) and the model acquisition section (15) acquire face information and model information from the same image (Im1). get.

According to this aspect, it is possible to acquire face information and model information even with a single imaging device (2) for obtaining an image (Im1).

In the monitoring system (1) according to the tenth aspect, in any one of the seventh to ninth aspects, the model information is a body part of the subject (H1) outside the image (Im1) for extracting the model information is included in at least one of the plurality of feature points (P1 to P7).

According to this aspect, there is no need to widen the angle of the imaging device (2) for the image (Im1), and for example, it is easy to improve the resolution of each part of the face (F1) in the image (Im1).

In the monitoring system (1) according to the eleventh aspect, in any one of the first to tenth aspects, the determination unit (16) determines the degree of stiffness of the upper body of the subject (H1) in the model information and the degree of stiffness of the subject (H1 ), the state of consciousness is determined based on at least one of the posture of the upper half of the body of

According to this aspect, since at least one of the degree of stiffness of the upper body of the subject (H1) and the posture of the upper body of the subject (H1) in the model information is used for determination, the reliability of determination of the state of consciousness is improved. easy to plan.

A mobile body (4) according to a twelfth aspect comprises the monitoring system (1) according to any one of the first to eleventh aspects, and a mobile body (41) on which the monitoring system (1) is mounted. .

According to this aspect, not only the face information about the state of each part of the face (F1) of the subject (H1), but also the model information about the posture model (M1) representing at least the posture of the upper body of the subject (H1). Based on this, the state of consciousness of the subject (H1) is determined. Therefore, compared to the case where the state of consciousness of the subject (H1) is determined only from the face information, it becomes easier to correctly determine the state of consciousness of the subject (H1). Therefore, for example, when the target person (H1) is simply looking down temporarily, it is difficult to make an erroneous determination that the target person (H1) is dozing off. As a result, according to the moving body (4), it is possible to make erroneous determination less likely to occur.

A monitoring method according to a thirteenth aspect includes a first acquisition process, a second acquisition process, and a determination process. The first acquisition process is a process of acquiring face information. The face information is information about the state of each part of the face (F1) of the subject (H1). The second acquisition process is a process of acquiring model information. The model information is information about a posture model (M1) that includes a plurality of feature points (P1 to P7) and represents at least the posture of the subject's (H1) upper body. The determination process is a process of determining the state of consciousness of the subject (H1) based on both face information and model information.

According to this aspect, not only the face information about the state of each part of the face (F1) of the subject (H1), but also the model information about the posture model (M1) representing at least the posture of the upper body of the subject (H1). Based on this, the state of consciousness of the subject (H1) is determined. Therefore, compared to the case where the state of consciousness of the subject (H1) is determined only from the face information, it becomes easier to correctly determine the state of consciousness of the subject (H1). Therefore, for example, when the target person (H1) is simply looking down temporarily, it is difficult to make an erroneous determination that the target person (H1) is dozing off. As a result, according to the monitoring method, it is possible to make erroneous determinations less likely to occur.

A program according to a fourteenth aspect is a program for causing one or more processors to execute the monitoring method according to the thirteenth aspect.

According to this aspect, not only the face information about the state of each part of the face (F1) of the subject (H1), but also the model information about the posture model (M1) representing at least the posture of the upper body of the subject (H1). Based on this, the state of consciousness of the subject (H1) is determined. Therefore, compared to the case where the state of consciousness of the subject (H1) is determined only from the face information, it becomes easier to correctly determine the state of consciousness of the subject (H1). Therefore, for example, when the target person (H1) is simply looking down temporarily, it is difficult to make an erroneous determination that the target person (H1) is dozing off. As a result, according to the above program, it is possible to make erroneous determination less likely to occur.

Various configurations (including modifications) of the monitoring system (1) according to Embodiments 1 and 2 can be embodied by a monitoring method or a program, without being limited to the above aspect.

The configurations according to the second to eleventh aspects are not essential to the monitoring system (1) and can be omitted as appropriate.

1 monitoring system 4 moving object 13 face information acquiring unit 14 face position acquiring unit 15 model acquiring unit 16 judging unit 41 moving object main body F1 face H1 target person Im1 image M1 posture model P1 to P7 feature points

Claims (12)

a face information acquisition unit that acquires face information about the state of each part of the face of the subject;
a model acquisition unit that acquires model information about a posture model that includes a plurality of feature points and represents at least the posture of the subject's upper body;
a determination unit that determines the subject's state of consciousness based on both the face information and the model information;
a face position acquisition unit that acquires face position information regarding at least one of the position and orientation of the face of the subject ;
The face position information is used for determination of the state of consciousness by the determination unit,
The determination unit changes a determination algorithm based on a correlation between the position of the face included in the face position information and the position of the face included in the model information.
Monitoring system. The determination unit determines the state of consciousness based on the face information if the reliability of the determination based on the face information is equal to or greater than a threshold, and if the reliability of the determination based on the face information is less than the threshold, determining the state of consciousness based on the model information;
A surveillance system according to claim 1 . the face information includes information about the condition of the eyelids of the subject;
3. The surveillance system according to claim 1 or 2. The determination unit changes a determination algorithm depending on whether the subject's blinking is not detected or when the subject's closed eyes are detected.
4. A surveillance system according to claim 3. both the face information and the model information are extracted from an image;
The monitoring system according to any one of claims 1-4. The information used by the determination unit for determination is extracted only from the image,
A surveillance system according to claim 5 . the face information acquisition unit and the model acquisition unit acquire the face information and the model information from the same image;
7. A surveillance system according to claim 5 or 6. the model information includes, in at least one of the plurality of feature points, a point corresponding to a part of the subject's body outside the image for extracting the model information;
A monitoring system according to any one of claims 5-7. The determination unit determines the state of consciousness based on at least one of a degree of stiffness of the upper body of the subject in the model information and a posture of the upper body of the subject.
A monitoring system according to any one of claims 1-8. A monitoring system according to any one of claims 1 to 9;
a mobile body on which the monitoring system is mounted;
Mobile. a first acquisition process for acquiring face information about the state of each part of the subject's face;
a second acquisition process for acquiring model information about a posture model that includes a plurality of feature points and represents at least the posture of the subject's upper body;
a determination process for determining the subject's state of consciousness based on both the face information and the model information;
a face position acquisition process for acquiring face position information regarding at least one of the position and orientation of the face of the subject;
The face position information is used to determine the state of consciousness in the determination process,
In the determination process, a determination algorithm is changed based on a correlation between the position of the face included in the face position information and the position of the face included in the model information.
Monitoring method. A program for causing one or more processors to execute the monitoring method according to claim 11.

Images