JP2007236488A - Vigilance degree estimating device, system, and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To increase the estimation precision when the estimation of the vigilance degree of a subject is performed by using movements of a plurality of kinds. <P>SOLUTION: This vigilance degree estimating device is equipped with means 120, 130 and 132 for acquiring the movement information of the subject through a means 110 which detects a plurality of kinds of movements of the subject, and a means 152 for keeping information regarding the priority in the various kinds of movements. The vigilance degree estimating device is also equipped with a means 150 for processing the movement information of the subject from the information regarding the priority, and a vigilance degree estimating means 160 for estimating the vigilance degree of the subject from the processing result of the movement information based on the information regarding the priority. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a wakefulness estimation apparatus, system, and method for estimating a wakefulness of a subject with high accuracy.

Conventionally, a near-infrared light source that illuminates the human eyeball, an imaging unit that images the human eyeball, an extraction unit that extracts a pupil region of the eyeball from an image captured by the imaging unit, and the extraction A wakefulness detection device characterized by comprising: a judging means for judging that the wakefulness state of a human is lowered when the blinking time and frequency are equal to or greater than a predetermined value from the shape change of the pupil region extracted by the means (For example, refer to Patent Document 1).
JP-A-6-270711

  By the way, there are various human movements correlated with sleepiness, but in many cases, the degree of arousal at which the movement is likely to appear varies depending on the type of movement. For example, yawns are likely to appear with a low arousal level, but frowning faces may appear with a relatively high level of arousal level. In addition, even the same type of movement may appear with different arousal levels. Furthermore, when the degree of arousal is different, the frequency of occurrence is often different even in the same movement. In addition, when two or more types of movements occur close in time, there is often a causal relationship between the combination of these actions and the arousal level. Therefore, in the case of a configuration in which the arousal level is estimated using a plurality of types of movements, the estimation accuracy of the awakening level of the subject can be improved if the various movements, the combination thereof, or the occurrence frequency cannot be appropriately evaluated. Can not.

  Therefore, an object of the present invention is to increase estimation accuracy in an arousal level estimation apparatus, system, and method that estimate arousal level of a subject using a plurality of types of movements.

In order to achieve the above object, the arousal level estimation device according to the first invention includes means for acquiring subject movement information via means for detecting a plurality of types of movements of the subject,
Means for holding information about the priority of various movements;
Means for processing subject movement information based on information about the priorities;
And awakening degree estimating means for estimating the awakening degree of the subject based on the processing result of the motion information based on the information related to the priority.

A second aspect of the present invention is the awakening level estimation device according to the first aspect,
The information related to the priority of each type of motion is generated based on a learning result of a correspondence relationship between the time of occurrence of various types of motion and the arousal level at that time. As a result, it is possible to perform an arousal level estimation process that matches the learning result of the occurrence status of various movements at each arousal level (two or more stages), so that the estimation accuracy of the arousal level can be increased.

The awakening level estimation device according to a third aspect of the present invention is a means for acquiring subject movement information via means for detecting a plurality of types of movements of the subject;
Means for holding information about arousal level in association with a combination of two types of movements;
When motion information indicating that two types of movements have occurred continuously within a predetermined time is acquired, the arousal level estimation is performed to estimate the arousal level of the subject based on information about the arousal level corresponding to the combination of the motions. Means. As a result, it is possible to appropriately evaluate the combination of the two types of actions and increase the estimation accuracy of the arousal level.

  According to a fourth aspect of the present invention, in the wakefulness level estimation device according to the third aspect of the invention, the information about the wakefulness level is based on a learning result of a correspondence relationship between the occurrence time of each combination of two types of movements and the wakefulness level at that time. It is characterized by being generated. As a result, it is possible to perform an arousal level estimation process adapted to the learning result of the occurrence state of various motion combinations at each arousal level (two or more stages), and thus it is possible to improve the estimation accuracy of the arousal level.

The arousal level estimation apparatus according to a fifth aspect of the present invention is a means for acquiring subject movement information via means for detecting a plurality of types of movements of the subject;
Means for calculating the frequency of occurrence of various movements based on movement information;
Means for holding a learning result of the frequency of occurrence of various movements at each arousal level;
And awakening degree estimating means for estimating the awakening degree of the subject on the basis of the correspondence between the calculated occurrence frequency of the movement and the learning result. As a result, it is possible to appropriately evaluate the frequency of occurrence of motion and increase the estimation accuracy of the arousal level.

  A wakefulness level estimation system according to a sixth aspect is characterized by comprising at least two wakefulness level estimation devices among the three wakefulness level estimation devices according to the first, third, and fifth aspects of the invention. In this case, any result may be prioritized and the final estimation result may be output, or all the results may be comprehensively evaluated and the final estimation result may be output.

The awakening level estimation method according to a seventh aspect of the present invention includes a step of acquiring subject movement information via means for detecting a plurality of types of movements of the subject,
A learning stage for learning the correspondence between the occurrence of various movements and the arousal level at that time,
When a movement is detected based on the movement information of the subject, the awakening level corresponding to the detected movement is extracted based on the learning result in the learning stage, and the extracted awakening level is used as the awakening level of the subject. And outputting a degree estimation result. As a result, it is possible to appropriately evaluate the causal relationship between the type of motion and the arousal level, and to increase the estimation accuracy of the arousal level.

The arousal level estimation method according to an eighth aspect of the present invention includes a step of acquiring subject movement information via means for detecting a plurality of types of movements of the subject,
A learning stage for learning the correspondence between the occurrence time of each combination of two types of movement and the arousal level at that time,
When two types of movements based on the movement information of the subject are detected close in time, the arousal level corresponding to the detected combination of movements is extracted based on the learning result in the learning stage. And outputting the extracted arousal level as an estimation result of the awakening level of the subject. As a result, it is possible to appropriately evaluate the causal relationship between the combination of two types of actions and the arousal level, and to increase the estimation accuracy of the arousal level.

The awakening level estimation method according to a ninth aspect of the present invention includes a step of acquiring subject movement information via means for detecting a plurality of types of movements of the subject,
Calculating the frequency of various movements based on the movement information;
A learning stage for learning the correspondence between the occurrence frequency of various movements and the arousal level at that time,
When a certain motion is detected based on the subject's motion information, based on the learning result in the learning stage, the awakening level corresponding to the calculated frequency of the motion is extracted, and the extracted awakening level is And a step of outputting as an estimation result of the awakening level of the subject. As a result, it is possible to appropriately evaluate the causal relationship between the frequency of occurrence of motion and the arousal level, and to increase the estimation accuracy of the arousal level.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to raise estimation precision in the arousal level estimation apparatus, system, and method which estimate a test subject's arousal level using multiple types of motion.

  The best mode for carrying out the present invention will be described below with reference to the drawings. Below, the case where a wakefulness estimation system is mounted in a vehicle and a test subject is a driver of a vehicle is demonstrated. However, the present invention can also be applied to applications other than vehicles.

  FIG. 1 is a system configuration diagram of an embodiment of a wakefulness estimation system according to the present invention. The arousal level estimation system 100 according to the present embodiment includes an image acquisition unit 110, an action tracking module 120, an action trigger generation module 130, an action database 132, an action frequency calculation module 140, an action prioritization module 150, a priority. A degree database 152, an action history database 154, and a wakefulness estimation module 160. A warning means 170 is connected to the arousal level estimation system 100.

FIG. 2 is a diagram showing a flow of main processing realized by the arousal level estimation system according to the present embodiment. The main processing realized by the arousal level estimation system according to this embodiment is as follows:
Step 10: Image acquisition processing by the image acquisition means 110;
Step 20: Action tracking processing by the action tracking module 120;
Step 30: Action trigger generation processing by the action trigger generation module 130;
Step 40: Action frequency calculation processing by the action frequency calculation module 140;
Step 50: Action prioritization processing by the action prioritization module 150;
Step 60: Arousal level estimation processing by the arousal level estimation module;
Step 70: The warning processing by the warning means 170 is included.

  Hereinafter, each process will be described in detail.

[Image acquisition processing]
In the image acquisition process 10, image acquisition by the image acquisition unit 110 is realized. The image acquisition unit 110 is, for example, a camera including a color or infrared sensitive CCD (charge-coupled device) sensor array. The image acquisition unit 110 is provided at an appropriate location of the vehicle so that the front surface of the driver (for example, the face portion from the front) can be captured, for example. For example, the image acquisition unit 110 is installed on a dashboard or a steering column of an instrument panel of a vehicle. The image acquisition unit 110 may acquire an image of the driver in real time and supply the image to the action tracking module 120, typically in a stream format of 30 fps (frame per second).

[Action tracking process]
The action tracking module 120 determines whether or not a predetermined action has occurred based on each image acquired by the image acquisition unit 110.

  FIG. 3 is a flowchart showing details of the action tracking process (step 20 in FIG. 2) realized by the action tracking module 120.

  In step 200, representative points (feature points) of the face are extracted from each image acquired by the image acquisition unit 110. This feature extraction method may be any appropriate method, and for example, a technique based on Active Appearance Model (AAM) may be used. An appropriate edge detection algorithm (eg, Sobel's edge detection algorithm) is then applied to determine the boundary between the facial features.

  In step 202, using the edges and feature points extracted as described above, the facial parts are separated as shown in FIG. 4, and the shape of the facial parts is extracted. FIG. 4 shows only a part of the face as a representative, and shows an image in which the eyebrows and eyes are extracted as parts.

  In step 204, a change in head orientation and part shape deformation are estimated. By this estimation process, three types of change parameters P, E, and M are derived.

The change parameter P includes a posture change parameter p _i that can represent a change mode of the orientation of the head, and is represented by P = {p _i }, i = 1,. Α is the number of posture change parameters. That is, the change parameter P is composed of a set of α posture change parameters. The posture change parameter may be represented by, for example, a rotation angle about three axes with the front direction of the face in a normal posture as one axis.

Change parameter E consists eyes change parameter e _j that can represent the shape variations of the eye _{region, E = {e j},} j = 1, represented by ... beta. Β is the number of eye change parameters. That is, the change parameter E is composed of a set of β eye change parameters. The eye change parameter may be, for example, a parameter indicating the size, area, width, height, etc. of the entire eye.

The change parameter M includes a mouth change parameter m _k that can represent the shape deformation mode of the mouth region, and is represented by M = {m _k }, k = 1,. Note that γ is the number of mouth change parameters. That is, the change parameter M is composed of a set of γ mouth change parameters. The mouth change parameter may be a parameter that represents, for example, the size, area, width, height, etc. of the entire mouth.

FIG. 5 shows an example of how the parameters p _i , e _j , and m _k change at each t second interval (typically, t = 1 second). In this way, by dividing the change parameter by a plurality of parts or orientations, various actions can be detected in a multifaceted manner.

[Action trigger generation processing]
The action trigger generation module 130 uses an action database 132 constructed in advance based on test results for a large number of unspecified subjects. The action database 132 stores change parameters related to face orientation and deformation for each action. In this example, predetermined different actions A _l (l = 1,...) Observed at an early stage of sleepiness are used as detection targets. The action to be detected is, for example, sigh, yawn, open eyes, mouth movements when making a frown.

The action trigger generation module 130 determines whether or not an action to be detected has occurred by matching the change parameters of the current posture, eye, and mouth category with values in the action database 132. If a match is found within a predetermined tolerance threshold, it is determined that the corresponding action has occurred and an action trigger data packet is generated along with the action information. As a simple example, for example, when a change mode of the parameter m _k as shown in FIG. 6A is obtained in a certain t seconds, the change mode is shown in the action database 132 as shown in FIG. 6B. To the same change mode related to the action A _i in FIG. 6C and the same change mode related to the action A _j in the action database 132 as shown in FIG. In this case, for example, matching within a predetermined tolerance threshold of the variant according to the action A _i is achieved, so that the action A _i is judged to have occurred.

  When the action trigger generation module 130 detects the occurrence of an action as described above, the action trigger generation module 130 generates an action trigger data packet. Action trigger data packets are generated at appropriate time intervals (in this example, every second).

FIG. 7 is a diagram illustrating an example of the data structure of the action trigger data packet. In the packet ID field, an action ID code that identifies an action determined to have occurred is written. Also, the action start time, duration, and action size are written. The magnitude of the action may be represented by an average value of the magnitudes of the parameters p _i , e _j and m _k . When the action generation ends, the continuation / end bit is set to OFF (zero). For example, when an action continues for 1 second, one action trigger data packet is generated.

[Action frequency calculation process]
The action frequency calculation module 140 calculates the frequency of occurrence of each action using the action trigger data packet. The action frequency calculation module 140 first puts an action trigger data packet in a section of T seconds (preferably 180 seconds) in a buffer and checks whether or not an action has occurred. For example, when N data packets indicating the occurrence of the action A _i are accumulated in the buffer, the occurrence frequency of the action A _i is represented by N / T. The calculated occurrence frequency of each action is recorded, and the buffer is refreshed every T. When the frequency is not calculated at all, it means that there is no action during the T seconds.

Action prioritization process
FIG. 8 is a table showing an example of a correspondence relationship between the arousal level and the state of the subject. In this example, as shown in FIG. 8, the arousal level is represented by six levels L1 to L6. That is, the arousal level L1 is the state in which sleepiness is strongest, and the awakening level L6 is in the state in which sleepiness is weakest. Details are shown in FIG. In addition, this invention is not limited to dividing into 6 steps in particular, You may classify awakening degree in another aspect.

FIG. 9 is a map (look-up table) showing the correspondence relationship between the priorities ε assigned to the actions A _l (l = 1,..., 17). In the example shown in FIG. 9, the priorities are the end stage of the arousal level L6 (illustrated by reference numeral AF), the initial stage of the arousal level L5 (shown by reference numeral BF) and the end stage, the initial stage and the end stage of the arousal level L4, and the arousal level L3. Are determined in consideration of the change process (transition) within the same arousal level, such as the initial and final stages of the first, the initial and final stages of the arousal level L2, and the initial level of the arousal level L1.

  FIG. 10 is a table showing the relationship between the priority ε and the awakening levels L1 to L6. In the example shown in FIG. 10, the initial stage and the final stage of each arousal level are each divided into five. The priority ε is expressed by ε = β × α using the values of the coefficients β and α, and is given to the time intervals divided into equal intervals within each arousal level. The coefficients β and α will be described with reference to FIGS.

  FIG. 11 is a conceptual diagram showing the change process of the arousal level in time series. In FIG. 11, a curve (wakefulness change curve) indicating a change state of the arousal level is shown on the time axis, and the region above the awakening level change curve represents the region IZ where sleepiness is strong, The region on the side represents the region RZ where drowsiness is weak.

The coefficient β is determined for each arousal level based on the time series of the arousal level as shown in FIG. For example, the coefficient β ₅ related to the final AF of the arousal level L5 and the coefficient β ₅ related to the initial BF of the arousal level L4 are on the time axis of the arousal level change curve part belonging to the arousal levels L4 to L5 as shown in FIG. This is represented by the area ratio between the upper region IZ _L5 and the lower region RZ _L5 partitioned by the curved portion in the section. That is, β ₅ = IZ _L5 / RZ _L5 .

FIG. 12 shows a coefficient β statistically derived based on a large number of unspecified test data and a wakefulness change curve. In FIG. 12, the coefficient β ₆ indicates the coefficient β related to the terminal AF of the wakefulness L6, the coefficient β ₅ indicates the coefficient β related to the terminal AF of the wakefulness L5 and the coefficient β related to the initial BF of the wakefulness L4. , Coefficient β ₄ indicates the coefficient β related to the end AF of the arousal level L4 and the coefficient β related to the initial BF of the arousal level L3, and the coefficient β ₃ indicates the coefficient β and the arousal level L2 related to the final AF of the arousal level L3. shows the coefficients according to the initial BF beta, factor beta ₂ shows a coefficient beta of the initial BF coefficient beta and alertness L1 according to the end AF of alertness L2. Each value of the coefficients β ₆ , β _5, etc. is used as each value of the coefficient β at each time interval within each arousal level in the table shown in FIG.

  FIG. 13 is a diagram showing an arrangement mode of the coefficients α in each time interval within each arousal level. In addition, as shown in FIG. 12, the time which stays at each arousal level is not constant actually. For this reason, the coefficient α is given by normalizing the time within each arousal level by, for example, 100 seconds. Further, as shown in FIG. 13, the coefficient α is given to a time interval (time position) obtained by dividing the time within each arousal level into 10 equal intervals. At that time, the coefficient α becomes the largest at the time position near the boundary between the initial stage and the final stage within the same degree of wakefulness, for example, the time interval near the boundary between the early stage and the final stage of the wakefulness L5 (in this example, 2 and 1.9), with the vicinity of the boundary as the center, the distance is set so as to decrease toward the other adjacent arousal level. The coefficient α is used together with the coefficient β to obtain the priority ε for each time interval after normalization, as shown in FIG. Therefore, the priority ε required for each time position within each arousal level is a different value.

The action prioritization module 150 learns the correspondence between the action occurrence state and the alertness levels L1 to L6 at that time by a test (sensory evaluation test) performed in a state where the alertness level of the subject is grasped, and the priority database 152 is constructed. Specifically, based on the arousal level change curve obtained by the sensory evaluation test, the above-described coefficients β _{1 to 6} and each coefficient α are calculated, and the priority ε and the arousal level L1 as shown in FIG. Build a correspondence with ~ L6. In addition, based on the type and timing of each action detected in the sensory evaluation test (awakening level when it occurs), a correspondence relationship between each action _Al and priority ε as shown in FIG. 9 is constructed. . For example, when the action A ₁₆ occurs at the initial time position (40 <t <50) of the arousal level L5 as shown in FIGS. 11 and 13, α = 2.0 and β = 0 by the above calculation method. Since .11, ε = 2.2. In this way, the map shown in FIG. 9 is generated (learned) based on the test result of the sensory evaluation test. In the map shown in FIG. 9, only values necessary for explanation are shown as an example, * indicates a certain value, and a blank indicates that the corresponding action was not detected in the sensory evaluation test. means.

  The priority database 152 may include a general-purpose database that is built in advance based on learning results for an unspecified number of subjects and a personal database that is built in advance based on learning results for specific subjects. In this case, prioritization is performed based on the general-purpose database until the subject is specified. After the subject is identified, prioritization may be performed based on the personal database, and the personal database may be updated. In addition, when the personal database for the subject has not yet been constructed, the personal database for the subject may be constructed and updated by learning during actual operation.

FIG. 14 shows a wakefulness change curve when the same subject is tested on different days (day1, day2). As shown in FIG. 14, the awakening degree change curve may be different due to the physical condition of the day. When constructing a personal database, considering this point, the characteristics of the wakefulness change curve are classified into several groups, and each group has a coefficient β (and finally a priority ε values, and you may learn the correspondence relation) of the priority ε for each action a _l.

  By the way, in practice, each action may occur alone, or a plurality of different actions may occur successively in time proximity.

  Therefore, the action prioritization module 150 according to the present embodiment evaluates the proximity of the occurrence times of two or more types of actions. Specifically, when two actions occur close in time, the action prioritization module 150 indicates an index value τ (hereinafter referred to as “proximity τ”) indicating the temporal continuity of these actions. Calculated).

FIG. 15 shows, as an example, a situation in which action A ₀₁ occurs at duration Δt1, and then action A ₀₃ occurs at duration Δt2 after time I. Action prioritization module 150, according to the time I from the end of the action _{A 01} and the commencement of the action _{A 03,} to determine the proximity τ about actions _{A 03.}

  FIG. 16 is a diagram illustrating a correspondence relationship between the time I and the proximity τ. As shown in FIG. 16, the proximity τ increases as the time I decreases. Such map data indicating the correspondence between the time I and the proximity τ is stored in a memory accessible by the action prioritization module 150.

FIG. 17 is a table showing the correspondence between two types of actions and their context. In FIG. 17, wakefulness levels L1 to L6 (including differences between BF and AF) corresponding to each combination of the six actions A _{01 to} A ₀₆ are shown. In addition to the difference between BF and AF, as shown in FIG. 10, the inside of each arousal level may be further subdivided to take into account the difference in each time position (ie, 40 <t <50). . In FIG. 17, the proximity τ is also mapped.

Similarly, the action prioritization module 150 learns the correspondence between two or more types of actions that occur in close proximity in time and the arousal levels L1 to L6 at that time by learning through a sensory evaluation test. At this time, the action prioritization module 150 performs learning in consideration of the occurrence order of two actions that occur successively. This is because there is often a causal relationship between the order of occurrence of the two actions and the arousal level at that time. For example, as shown in FIG. 15, when the action A ₀₃ occurs within 5 to 10 seconds after the completion of the action A _{01 and} the awakening level at that time is the terminal AF of the awakening level L5, The combination is associated with the final AF of the arousal level L5 (proximity τ = 90 is also given). In this way, the map shown in FIG. 17 is generated (learned) based on the test result of the sensory evaluation test. In the map shown in FIG. 17, only values necessary for explanation are shown as an example, * indicates a certain value, and a blank indicates that the combination of the corresponding actions was not detected in the sensory evaluation test. Means that.

  Similarly, the action prioritization module 150 constructs the action history database 154 by learning through a sensory evaluation test. In the action history database 154, for each type of action detected in the sensory evaluation test, the correspondence between the action frequency and the awakening level L1 to L6 at that time is learned for each type. The action history database 154 may be classified and constructed for each subject. This is because there is an individual difference in the correspondence between the action frequency and the awakening levels L1 to L6 at that time (see FIG. 18).

FIG. 18 is a graph showing, as an example, a correspondence relationship between the action frequency related to the action A ₀₁ and the arousal levels L1 to L6. In FIG. 18, plot points representing the correspondence relationship are plotted for each subject with respect to three subjects (subjects 1, 2, and 3). In addition, plot points representing the average correspondence are plotted based on the average value of these data. In the example shown in FIG. 18, the difference between the initial stage and the final stage in each awakening level and the difference in time position finer than that are not taken into consideration, but these can be taken into consideration. For example, it is possible to learn the correspondence between the action frequency and the initial stage and the final stage of each arousal level. The threshold value Th shown in FIG. 18 will be described later in connection with the arousal level estimation process.

  FIG. 19 is a flowchart showing details of the action prioritization process (step 50 in FIG. 2) realized by the action prioritization module 150. As described above, the action prioritization module 150 is supplied with an action trigger data packet each time an action is detected.

  In step 500, the action prioritization module 150 decodes and decodes the action trigger data packet supplied from the action trigger generation module 130, and the type of action (detected action) occurring at the present time, the start time, etc. Is identified. The action type and start time specified in this way are stored in the action history database 154. In this way, history information of actions that have occurred is accumulated in the action history database 154.

  In step 510, the action prioritization module 150 uses the action history database 154 to search for other actions detected immediately before the action detected this time, and based on the map shown in FIG. Depending on the time I between these two actions, the proximity τ is derived. It should be noted that if there is no other action detected before the action detected this time, or if another action is detected even if it exists, for example, 50 seconds or more before, the proximity degree τ may be zero.

  In step 520, the action prioritization module 150 grasps the currently estimated wakefulness of the subject based on the wakefulness estimation result supplied from the wakefulness estimation module 160 as will be described later.

In step 530, the action prioritization module 150 uses the learned priority database 152 to derive the priority ε corresponding to the action detected this time. That is, the action prioritization module 150 determines the priority ε of the action detected this time with reference to the map shown in FIG. At this time, the priority ε assigned to the action detected this time is determined in consideration of the currently estimated degree of awakening of the subject. This is because there are actions that can appear at various arousal levels as shown in FIG. For this reason, the action prioritization module 150 adopts the priority at the alertness level closest to the currently estimated subject alertness level on the assumption that the alertness level does not change significantly in a short time. For example, there is a plurality of priorities ε (0.17, 0.20,...) For the action A ₀₁ , but the time position of 40 <t <50 in the initial BF of the arousal level L5 is When it is estimated as the current awakening level of the subject, the priority “0.20” in the awakening level close to the time position is selected. When the time position of each candidate is close to the time position related to the currently estimated degree of arousal of the subject, either one may be selected based on other conditions, or The priority (0.22 in this example) with respect to the time position related to the arousal level of the subject that is currently estimated may be maintained assuming that any of the selections is impossible. Moreover, at the time of detecting an action for the first time, the arousal level L6 may be the currently estimated awakening level (initial value) of the subject. That is, it may be initially assumed that the degree of arousal of the subject is high.

  In step 540, the action prioritization module 150 outputs the proximity τ and the priority ε derived in steps 510 and 530 described above to the arousal level estimation module 160.

[Arousal level estimation processing]
FIG. 20 is a flowchart showing details of the action tracking process (step 60 in FIG. 2) realized by the arousal level estimation module 160. The awakening level estimation module 160 calculates the priority ε and the proximity τ given to the action by the action prioritizing module 150 every time an action is detected, and the action frequency as described above. The occurrence frequency of the action calculated by the module 140 (that is, the occurrence frequency of the action within the past T seconds from the present time) is supplied as needed. The following processing from step 600 to step 612 is executed every time a new action is detected.

  In step 600, the arousal level estimation module 160 determines whether or not the proximity τ is greater than or equal to a predetermined value (50 in this example). That is, it is determined whether or not another action has occurred in a section that is a predetermined time (30 seconds before this example) before the start time of the action detected this time. When the proximity τ is less than 50, the process proceeds to step 602, and when the proximity τ is 50 or more, the process proceeds to step 604.

  In step 602, the arousal level estimation module 160 refers to the map shown in FIG. 10 and estimates the awakening level of the subject based on the priority ε from the action prioritization module 150. For example, when the priority ε assigned to the action detected this time is 0.17, the arousal level estimation module 160 refers to the map shown in FIG. It is estimated that “20 <t <30 time position in the final AF of the arousal level L6”.

In step 604, the arousal level estimation module 160 refers to the map shown in FIG. 17 and estimates the arousal level corresponding to the combination of the two actions as the awakening level of the subject. For example, when the action detected this time is action A ₀₁ and the action detected within 30 seconds before that is action A ₀₅ , the arousal level estimation module 160 refers to the map shown in FIG. Then, the awakening level of the subject is estimated as “the initial BF of the awakening level L3”. Note that, as described above, when the map shown in FIG. 17 has been learned up to the time position, the corresponding time position in the initial BF of the arousal level L3 is taken as the estimation result.

  In step 606, the wakefulness estimation module 160 determines whether or not the wakefulness estimated as described above has changed with respect to the previously estimated wakefulness. In this case, the change in wakefulness is not a change in time position within the wakefulness, but the wakefulness L6, L5,. . . , And the change between L1. However, instead of this, the adjacent arousal level, such as a change from the final AF of the arousal level L6 to the final AF of the awakening level L5, a change beyond the final AF level of the awakening level L4 of the final AF of the arousal level L5, etc. It may be a change in the last period. If the estimated wakefulness level has changed with respect to the previously estimated wakefulness level, the process proceeds to step 608; otherwise, the process proceeds to step 610.

  In step 608, the arousal level estimation module 160 is supplied from the action frequency calculation module 140 with the occurrence frequency of various actions during the period until the arousal level changes (that is, in the state of the awakening level before the change). Calculation is performed based on the history of calculation results. The arousal level estimation module 160 records the occurrence frequency of various actions in the action history database 154 in association with the awakening level before the change. Thereby, learning of the action history database 154 during actual operation is realized.

In step 610, the arousal level estimation module 160 determines whether or not the occurrence frequency of the action detected this time has exceeded a predetermined threshold value. The predetermined threshold may be determined according to the currently estimated arousal level. For example, in a situation where currently alertness L6 is estimated, relative to the frequency of occurrence of the action A _01, the threshold Th may be applied as shown in FIG. 18. In this case, when the action detected this time is action A ₀₁ and the occurrence frequency exceeds the threshold Th, the arousal level estimation module 160 estimates that the arousal level has decreased from L6 to L5. Moreover, as is clear from the characteristics shown in FIG. 18, when the frequency of occurrence of the action A ₀₁ is focused on larger projects in alertness L4, the frequency of occurrence of the action A ₀₁ exceeds much larger threshold value Th ( For example, the wakefulness level estimation module 160 may estimate that the wakefulness level has decreased from L6 to L4 (when the occurrence frequency (learned value) vicinity corresponding to the wakefulness level L4 in FIG. 18 is reached). Moreover, you may set a different threshold value for every test subject based on the learning result classified and learned for every test subject. As described above, when the characteristics map shown in FIG. 18 is learned up to the initial BF and final AF, thresholds corresponding to the respective sections may be prepared.

  In step 612, the arousal level estimation module 160 outputs the arousal level estimated as described above to the warning unit 170 as the estimation result of the awakening level of the subject.

  By the way, in the case of the configuration in which the arousal level is estimated using a plurality of types of actions as in the present embodiment, the arousal level can be estimated at a fine level, so it is attacked by weak sleepiness that is difficult to detect. The state can be detected. That is, it is possible to detect the initial stage of a decrease in the arousal level before the subject falls into a state of being attacked by strong sleepiness. However, on the other hand, if each of these actions or each combination of these actions cannot be evaluated appropriately, the accuracy of estimation of the arousal level of the subject cannot be increased. As mentioned above, this often appears with different arousal levels even for the same action, and there are often some causal relationships between each combination of actions and the arousal level. This is because there are many cases in which the frequency of actions that occur differs depending on the degree of arousal.

  On the other hand, by using the learned priority database 152, the priority ε corresponding to each detected action and the arousal level based on it are determined based on the learning result of the correspondence relationship between the various actions and the awakening level. Therefore, each action can be evaluated appropriately. In addition, based on the learning result of the correspondence relationship between each combination of actions and the arousal level using the learned priority database 152, the arousal level of the subject when multiple types of actions occur in close proximity in time Therefore, each combination of actions can be evaluated appropriately. Furthermore, it is possible to appropriately evaluate the occurrence frequency of various detected actions based on the learning result of the correspondence relationship between the action frequency and the arousal level. As a result, according to the present embodiment, it is possible to accurately estimate the arousal level.

  Here, a specific example of the arousal level estimation process executed by the arousal level estimation module 160 and the action prioritization module 150 during actual operation (≠ learning) will be described.

Here, the maps shown in FIGS. 9 and 10 are used, and as shown in FIG. 21, action A ₀₆ , action A ₁₄ , action A ₀₂ , action A ₀₉ , action A _04, and action A ₁₆ are single-shot. Are detected in order. It is assumed that the proximity τ to these actions is less than 50, and the frequency of occurrence of each action does not exceed the threshold value.

When action A ₀₆ is detected as the first action, it is assumed that the current awakening level is L6, so that priority ε = 0.17 is given by the action prioritizing module 150 according to the map shown in FIG. Is done. As a result, according to the map shown in FIG. 10, the wakefulness estimation module 160 outputs “20 <t <30 time position in the final AF of the wakefulness L6” as the estimation result.

Next, when the action A ₁₄ is detected, it is estimated that the current wakefulness is “20 <t <30 time position in the final AF of the wakefulness L6”, and therefore, according to the map shown in FIG. The action prioritization module 150 gives the priority ε = 0.14. As a result, according to the map shown in FIG. 10, the wakefulness estimation module 160 outputs “30 <t <40 time position in the final AF of the wakefulness L6” as the estimation result.

Next, when the action A ₀₂ is detected, it is estimated that the current wakefulness is “30 <t <40 time position in the final AF of the wakefulness L6”, and therefore, according to the map shown in FIG. The priority ε = 0.13 is given by the action prioritizing module 150. As a result, according to the map shown in FIG. 10, the arousal level estimation module 160 outputs “a time position of t <10 in the initial BF of the arousal level L5” as an estimation result.

Next, when the action A ₀₉ is detected, it is estimated that the current arousal level is “time position of t <10 in the initial BF of the arousal level L 5”, so that the action is performed according to the map shown in FIG. The prioritization module 150 gives priority ε = 0.65. As a result, according to the map shown in FIG. 10, the wakefulness level estimation module 160 outputs “20 <t <30 time position in the final AF of the wakefulness level L5” as an estimation result.

Next, when the action A ₀₄ is detected, it is estimated that the current wakefulness is “20 <t <30 time position in the final AF of the wakefulness L5”, and therefore, according to the map shown in FIG. The priority ε = 0.69 is given by the action prioritization module 150. As a result, according to the map shown in FIG. 10, the wakefulness estimation module 160 outputs “20 <t <30 time position in the initial BF of the wakefulness L4” as the estimation result.

Next, when the action A ₁₆ is detected, it is estimated that the current wakefulness is “20 <t <30 time position in the initial BF of the wakefulness L4”, and therefore, according to the map shown in FIG. The action prioritization module 150 gives the priority ε = 1.1. As a result, according to the map shown in FIG. 10, the wakefulness estimation module 160 outputs “40 <t <50 time position in the final AF of the wakefulness L4” as the estimation result.

  As described above, even when each action occurs with a small frequency, the fluctuation process of the arousal level of the subject can be finely estimated by the arousal level estimation process according to the present embodiment.

[Warning]
The warning means 170 outputs a voice or video to the subject when, for example, a decrease in the arousal level is estimated based on the estimation result of the arousal level by the arousal level estimation module 160. The alarm output mode can be various, and when the subject is a driver, vibration is generated by a vibrating body embedded in the seat, air is blown out from an air hole provided in the seat (seat), and arousal is promoted An odor may be generated, the temperature of the steering wheel may be changed, or a warning sign may be output in the meter. Moreover, the warning means 170 may change the output mode of a warning finely according to the arousal level.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  As described above, the present invention can be used in any application in which it is useful to estimate a subject's arousal level, including an application that estimates a driver's arousal level in a vehicle.

It is a system configuration | structure figure of one Example of the arousal level estimation system by this invention. It is a figure which shows the flow of the main processes implement | achieved by the arousal level estimation system by a present Example. 3 is a flowchart showing details of an action tracking process (step 20 in FIG. 2) realized by an action tracking module 120. It is a figure which shows the image from which the eyebrows and eye part of the face were extracted as parts. An example of how the parameters p _i , e _j , and m _k change at each t-second interval (typically, t = 1 second) is shown. It is a figure which shows an example of an action generation | occurrence | production determination method. It is a figure which shows an example of the data structure of an action trigger data packet. It is a table | surface figure showing an example of the correspondence of arousal level and a test subject's state. It is a figure which shows an example of the map which shows the priority (epsilon) provided with respect to each action _Al (l = 1, ..., 17). It is a figure which shows an example of the map showing the relationship between priority (epsilon) and arousal levels L1-L6. It is a conceptual diagram which shows the change process of an arousal level in time series. It is a figure which shows each coefficient (beta) based on the arousal level change curve statistically derived | led-out based on the unspecified many test data. It is a figure which shows the arrangement | sequence aspect of the coefficient (alpha) in each time interval within each arousal level. It is a figure which shows the arousal level change curve at the time of testing on the same test subject on a different day. Other actions A _j immediately after the occurrence of the action A _i is a diagram showing a situation that occurred. It is a figure which shows an example of the map which shows the correspondence of time I and proximity degree (tau). It is a figure which shows an example of the map which shows the correspondence with the kind of two actions, and its context. Action frequencies for Action _{A 01} and is a graph showing a relationship between a degree of awakening L1 to L6. 3 is a flowchart showing details of an action prioritization process (step 50 in FIG. 2) realized by an action prioritization module 150. 5 is a flowchart showing details of an action tracking process (step 60 in FIG. 2) realized by the arousal level estimation module 160. It is explanatory drawing of the specific example of an arousal level estimation process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Arousal degree estimation system 110 Image acquisition means 120 Action tracking module 130 Action trigger generation module 132 Action database 140 Action frequency calculation module 150 Action prioritization module 152 Priority database 154 Action history database 160 Awakening degree estimation module 170 Warning means

Claims (9)

Means for acquiring subject movement information via means for detecting a plurality of types of movements of the subject;
Means for holding information about the priority of various movements;
Means for processing subject movement information based on information about the priorities;
A wakefulness level estimation device comprising: wakefulness level estimation means for estimating a wakefulness level of a subject based on a processing result of motion information based on information relating to the priority level.   The arousal level estimation apparatus according to claim 1, wherein the information related to the priority of each type of motion is generated based on a learning result of a correspondence relationship between the occurrence time point of various types of motion and the awakening level at that time. Means for acquiring subject movement information via means for detecting a plurality of types of movements of the subject;
Means for holding information about arousal level in association with a combination of two types of movements;
When motion information indicating that two types of movements have occurred continuously within a predetermined time is acquired, the arousal level estimation is performed to estimate the arousal level of the subject based on information about the arousal level corresponding to the combination of the motions. And a waking degree estimation device.   The wakefulness level estimation device according to claim 3, wherein the information about the wakefulness level is generated based on a learning result of a correspondence relationship between the occurrence time of each combination of two types of movements and the wakefulness level at that time. Means for acquiring subject movement information via means for detecting a plurality of types of movements of the subject;
Means for calculating the frequency of occurrence of various movements based on movement information;
Means for holding a learning result of the frequency of occurrence of various movements at each arousal level;
A wakefulness estimation device, comprising: wakefulness estimation means for estimating the wakefulness of a subject based on a correspondence relationship between the calculated frequency of movement and the learning result.   A wakefulness level estimation system comprising at least two wakefulness level estimation devices among the three wakefulness level estimation devices according to claim 1, 3 and 5. Obtaining movement information of the subject through means for detecting a plurality of types of movement of the subject;
A learning stage for learning the correspondence between the occurrence of various movements and the arousal level at that time,
When a movement is detected based on the movement information of the subject, the awakening level corresponding to the detected movement is extracted based on the learning result in the learning stage, and the extracted awakening level is used as the awakening level of the subject. And a step of outputting as a degree estimation result. Obtaining movement information of the subject through means for detecting a plurality of types of movement of the subject;
A learning stage for learning the correspondence between the occurrence time of each combination of two types of movement and the arousal level at that time,
When two types of movements based on the movement information of the subject are detected close in time, the arousal level corresponding to the detected combination of movements is extracted based on the learning result in the learning stage. And outputting the extracted arousal level as an estimation result of the awakening level of the subject. Obtaining movement information of the subject through means for detecting a plurality of types of movement of the subject;
Calculating the frequency of various movements based on the movement information;
A learning stage for learning the correspondence between the occurrence frequency of various movements and the arousal level at that time,
When a certain motion is detected based on the subject's motion information, based on the learning result in the learning stage, the awakening level corresponding to the calculated frequency of the motion is extracted, and the extracted awakening level is And a step of outputting as a result of estimating the degree of arousal of the subject.

Images