JP2022018451A - State determination device, state determination system, and control method - Google Patents

Abstract

To accurately determine the state of a person.SOLUTION: A state determination device (1C or 1D) comprises: a data acquisition unit (11) for acquiring sensor information from a group of sensors (2); an explanatory variable selection unit (71) which selects explanatory variables highly correlated with variation of an identified crew member into a specific state; a plurality of explanatory variable calculation units (17) which calculate the explanatory variables selected by the explanatory variable selection unit; score calculation unit (18) which calculates a score about each explanatory variable; and a score determination unit (19) which determines that the crew member is in the specific state, in a case where a total score is a predetermined threshold value or more.SELECTED DRAWING: Figure 11

Description

The present disclosure relates to a state determination device, a state determination system, and a control method.

Conventionally, there is a technique for determining the human condition from biological information such as pulse. For example, in Patent Document 1, the degree of arousal of a human being is calculated by comparing a combination of codes of eigenvector coefficients based on a human physiological index with a combination of the codes at the time of awakening stored in advance. The technology is disclosed.

Japanese Unexamined Patent Publication No. 2006-247055

However, there are individual differences in the values of physiological indicators such as respiration and heart rate. Therefore, the technique described in Patent Document 1 may not be able to accurately determine the human condition.

One aspect of the present disclosure is in view of the above problems, and an object of the present disclosure is to accurately determine the human condition.

In order to solve the above-mentioned problems, the state determination device according to one aspect of the present invention acquires sensor information about the occupant of the moving body from one or more sensor devices arranged in the internal space of the moving body. A unit, an explanatory variable selection unit that selects, as an explanatory variable, a state index value having a high correlation with the change of the occupant to a specific state from among a plurality of state index values indicating the identified states of the occupant, and the above-mentioned unit. With respect to the plurality of index value calculation units that calculate the state index values selected by the explanatory variable selection unit based on at least a part of the sensor information, and the state index values calculated by each of the plurality of index value calculation units. When the score calculation unit that calculates the score indicating the possibility that the occupant is in a specific state and the total score obtained by summing the scores are equal to or higher than a predetermined score threshold, it is determined that the occupant is in a specific state. It is characterized by having a score determination unit and a score determination unit.

According to the above configuration, a state index value having a high correlation with the change of the occupant to a specific state is selected, a score of the state index value is calculated, and a final state determination result is obtained based on the score. Can be given. Therefore, the human condition can be determined more accurately.

In the state determination device, the score calculation unit may calculate the score based on a mathematical formula showing a correlation between the state index value of the identified occupant and the probability of being in the specific state. .. According to the above configuration, the human condition can be determined more accurately.

The state determination device has a receiving unit that receives collected data from the management device, and a state threshold value that is a threshold value for determining whether or not the state of the occupant is the specific state based on the collected data. , The state threshold value calculation unit calculated for each of the state index values selected by the explanatory variable selection unit, and the state determination for determining whether or not the state index value is equal to or higher than the state threshold value corresponding to the state index value. The collected data may include, and the collected data includes the state index value of the occupant of the other moving body calculated in the other moving body, and the other moving determined in the other moving body. A plurality of data associated with the state of the occupant of the body may be included, and the score calculation unit calculates a score for the state index value determined by the state determination unit to be equal to or higher than the state threshold value. You may.

According to the above configuration, the state determination device is now according to the state index value of another occupant calculated by the other mobile body and the result of the state determination of the other occupant by the state index value. Calculate the state threshold value that is the standard for the state judgment performed from. In this way, since the determination score and the total score are calculated based on the accurately determined determination result, the state determination device can more accurately determine the human condition.

In the state determination device, the sensor information may be data indicating the biological beat number, and the index value calculation unit specifies the state index value using the sensor information in a predetermined first period. You may.

According to the above configuration, the human condition can be determined more accurately based on the data of the biological beat count in the predetermined first period.

The state determination device determines whether or not the state of the occupant is a specific state based on at least a part of the sensor information before the index value calculation unit calculates the state index value. It may be provided with a sub-state determination unit.

According to the above configuration, until the index value calculation unit obtains the data of the biological beat number for a period (that is, a predetermined first period) for which the state index value can be calculated, the sub-state determination unit is used. The condition of the occupant can be determined. Therefore, the human condition can be determined more quickly.

The state determination device determines whether or not the occupant is in a specific state based on the state index value or the value of the intermediate result when calculating the state index value from the sensor information. The average determination unit may include a unit, and the average determination unit may be used when the state index value or the intermediate result value is equal to or higher than the average value of the occupant's state index values in the past second period, or in the past. It may be determined that the occupant is in a specific state when it is equal to or more than the average value of the intermediate result values of the occupant in the second period.

If the state index value calculated by the index value calculation unit is equal to or greater than the average value of the occupant's own state index values calculated in the past second period or the average value of the intermediate results, the occupant's condition is the past average. There is a high possibility that it is different from the normal state, that is, the normal state. According to the above configuration, the state determination device can determine that the occupant has reached a specific state when the occupant's state is likely to be different from the normal state.

In the state determination device, the state threshold value calculation unit is a value equal to or higher than the median value of the state index values calculated in the other mobile body, and the occupant is in a specific state among the plurality of data. A value in which half or more of the data determined to be present may be used as the state index value in the moving body. According to the above configuration, the human condition can be determined more accurately.

The state determination device may include a notification unit that notifies a warning to the occupant via an output device when the occupant is determined to be in a specific state. According to the above configuration, it is possible to notify the occupant that the occupant is in a specific state.

In order to solve the above problems, the state determination system according to one aspect of the present invention includes the state determination device and the sensor device. According to the above configuration, the same effect as that of the state determination device is obtained.

In order to solve the above-mentioned problems, the control method of the state determination device according to one aspect of the present invention obtains sensor information about the occupant of the moving body from one or more sensor devices arranged in the internal space of the moving body. An explanatory variable selection step that selects, as an explanatory variable, a state index value having a high correlation with the change of the occupant to a specific state among a plurality of state index values indicating the state of the identified occupant and the acquisition step to be acquired. With respect to the index value calculation step for calculating the state index value selected in the explanatory variable selection step and the state index value calculated in the index value calculation step based on at least a part of the sensor information. When the score calculation step for calculating the score indicating the possibility that the occupant is in a specific state and the total score obtained by summing the scores are equal to or higher than the predetermined score threshold, it is determined that the occupant is in the specific state. It is characterized by including a score determination step. According to the above configuration, the same effect as that of the state determination device is obtained.

According to one aspect of the present disclosure, the human condition can be accurately determined.

It is a block diagram which shows the main part structure of the state determination system which concerns on Embodiment 1. It is a figure which shows schematically the calculation method of the 2nd threshold value in a graph. It is a figure which shows the data structure of a state log. It is a flowchart which shows the flow of the process in the state determination apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the flow of the process of the 1st determination. It is a flowchart which shows the flow of the process of the 2nd determination. It is a block diagram which shows the main part structure of the state determination system which concerns on Embodiment 2. It is a flowchart which shows the flow of processing in the state determination apparatus which concerns on Embodiment 2. It is a flowchart which shows the flow of the process of the 3rd determination. It is a graph which showed the appearance frequency of the RRI difference of a certain occupant in the 2nd period. It is a block diagram which shows the main part structure of the state determination system which concerns on Embodiment 3. It is a figure which shows the data structure of the explanatory variable data. It is a graph which shows the total threshold value, the individual threshold value, and the score expression of the explanatory variables which do not correlate with the state change of the occupant. It is a graph which shows the total threshold value, the individual threshold value, and the score expression of the explanatory variables which correlate with the state change of an occupant. It is a flowchart which shows the flow of the process in the state determination apparatus which concerns on Embodiment 3. It is a figure which shows typically the selection of the explanatory variable used for the state determination of a certain occupant, and the result of the state determination using the explanatory variable. It is a block diagram which shows the main part structure of the state determination system which concerns on Embodiment 4.

[Embodiment 1]
≪System overview≫
The state determination system 100 according to the present embodiment is a system for determining the state of an occupant staying in the internal space of a moving body. Here, the "state" refers to a human's physiological or emotional state. For example, the state determination system 100 can determine whether or not the occupant is drowsy, whether or not he is angry, whether or not he is in an excited state, and the like as the "state" of the occupant. In the present embodiment, as an example, the state determination system 100 for determining the presence or absence of drowsiness of an occupant will be described with reference to FIGS. 1 to 6.

≪Main part composition≫
FIG. 1 is a block diagram showing a configuration of a main part of the state determination system 100 according to the present embodiment. The state determination system 100 includes a state determination device 1A, a sensor group 2, a storage device 3, a server (management device) 4, and an output device 5. The storage device 3 is not an essential configuration.

In the example of FIG. 1, the state determination device 1A, the sensor group 2, the storage device 3, and the output device 5 are mounted on the moving body 6. Although only the internal structure of one moving body 6 is described in detail in FIG. 1 for simplification, all of the moving bodies 6 shown in FIG. 1 have the same configuration. Each sensor included in the sensor group 2 and the state determination device 1A are communicably connected to each other. The storage device 3 and the state determination device 1A are communicably connected to each other. The state determination device 1A and the server 4 are connected so as to be able to communicate with each other. As shown in FIG. 1, the server 4 communicates with a plurality of mobile bodies 6.

(Mobile 6)
The moving body 6 is a moving body having a space inside, such as an automobile, a train, and an aircraft. Hereinafter, when the term is simply referred to as "internal space", it means the space inside the moving body 6. Also, the person who stays in the interior space is simply called the "occupant".

(Sensor group 2)
The sensor group 2 is a group of sensors for measuring sensor information. Here, the "sensor information" indicates all the information about the occupant that can be acquired by the sensor group 2. For example, the sensor information may be data on the appearance of the occupant, such as the movement of the entire body or a part of the occupant, as well as facial expressions. Further, the sensor information may be information indicating vital data of the occupant such as pulse, respiration, body temperature, blood pressure, consciousness, and reflex. In the present embodiment, the sensor information indicates the data of the captured image of the camera (sensor device) 21 and the measurement data of the pulse sensor (sensor device) 22.

The sensor group 2 is composed of one or more sensors arranged in the space inside the moving body 6. Various sensors are connected to the state determination device 1A by wire or wirelessly, and the sensor information acquired by each is transmitted to the state determination device 1A. The type of sensor included in the sensor group 2 is not particularly limited. For example, the sensor group 2 includes a camera 21 and a pulse sensor 22 as output devices.

The camera 21 captures at least a part of the occupant's body. The body referred to here includes the face of the occupant. The captured image of the camera 21 is transmitted to the state determination device 1A as sensor information. The captured image of the camera 21 may be a still image or a moving image. Hereinafter, the "image" and the "photographed image" refer to both a still image and a moving image. When the camera 21 shoots a still image, the shooting time interval is not particularly limited, but it is preferable to shoot at a time interval such that the movement of the occupant can be tracked in real time. The camera 21 starts shooting at a predetermined timing, and thereafter continuously shoots the occupant. The predetermined timing may be, for example, the timing when the occupant is seated in the seat of the moving body 6.

Further, the specific configuration of the camera 21 and the arrangement position in the predetermined space are not particularly limited. For example, when the moving body 6 is a passenger car, the camera 21 may be arranged in front of the seat of the passenger car at a position and an angle so that the face of a person sitting in the seat can be seen. As a result, the camera 21 can acquire an image of the face of the driver or passenger of the passenger car captured from the front. In other words, the camera 21 can capture an image capable of identifying the movement of the driver's or passenger's eyeball, the movement of the eyelids, the facial expression, and the like.

The pulse sensor 22 is a sensor that measures a human pulse rate. The pulse sensor 22 may be, for example, an electrode-type pulse sensor arranged on an armrest portion of the seat of the moving body 6. Further, the pulse sensor 22 may be a radio wave type pulse sensor built in the seat of the moving body 6. More specifically, for example, the pulse sensor 22 may be a vital sensor that detects a heartbeat signal from a signal of a millimeter wave radar. The measurement data of the pulse sensor 22 is transmitted to the state determination device 1A as sensor information. The pulse sensor 22 starts measuring the pulse at a predetermined timing, and thereafter continuously measures the pulse of the occupant. The predetermined timing may be, for example, the timing when the occupant is seated in the seat of the moving body 6.

The sensor group 2 may include a sensor that measures a biological beat rate other than the pulse, such as a heart rate. In addition, the sensor group 2 may include a sensor for measuring vital data such as respiration, body temperature, blood pressure, consciousness, and reflex. The measurement data of the sensor is also transmitted to the state determination device 1A as sensor information.

In the internal space of the moving body 6, a plurality of sensors of the same type may be arranged at different positions. For example, when the moving body 6 is a passenger car, one or more cameras 21 may be arranged in front of each seat of the passenger car. This makes it possible to acquire an image of each person, for example, when a person sits in each seat. Further, one or more pulse sensors 22 may be arranged in each seat of the passenger car. This makes it possible to measure the pulse of each seat, that is, each person.

(Status determination device 1A)
The state determination device 1A determines the state of the occupant based on the sensor information acquired from the sensor group 2 and the collected data acquired from the server 4. The details of the collected data will be described later. The state determination device 1A includes a data acquisition unit (acquisition unit) 11, a first determination unit (secondary state determination unit) 12, a communication unit (reception unit) 13, a second determination unit 14, and a notification unit 15. Has.

The state determination device 1A according to the present embodiment performs a main state determination using the measurement data of the pulse sensor 22. However, in order to accurately identify the occupant's state from the occupant's pulse transition, it is desirable to measure the pulse for a certain period of time (for example, 5 minutes) using the pulse sensor 22.

Therefore, in the state determination device 1A according to the present embodiment, the state determination is performed in a shorter measurement time than the pulse sensor 22 during the period from the start of measurement of the pulse sensor 22 to the sufficient accumulation of the measurement data of the pulse sensor 22. The state is judged by another possible judgment method. Specifically, the state determination device 1A determines whether or not the occupant is drowsy by using the captured image of the camera 21.

Hereinafter, in the state determination device 1A, the state determination first performed after the start of measurement (or imaging) of the sensor group 2 is referred to as a first determination. Further, hereinafter, in the state determination device 1A, the state determination performed after the first determination is referred to as a second determination. The first determination is an additional configuration for determining the state at an earlier stage. Therefore, the configuration according to the first determination is not an essential configuration in the present embodiment.

The data acquisition unit 11 acquires the sensor information of the moving body 6 from the sensor group 2. The data acquisition unit 11 outputs at least a part of the acquired sensor information to the first determination unit 12. Further, the data acquisition unit 11 outputs at least a part of the acquired sensor information to the second determination unit 14. In the present embodiment, the data acquisition unit 11 outputs the captured image of the camera 21 to the first determination unit 12 and the pulse measurement data of the pulse sensor 22 to the second determination unit 14. The data acquisition unit 11 may continuously acquire sensor information from the sensor group 2 at a predetermined time interval or in real time.

The communication unit 13 communicates between the state determination device 1A and the server 4. For example, the communication unit 13 receives the collected data from the server 4 and outputs it to the second determination unit 14. Further, for example, the communication unit 13 transmits the state log 32 input from the second determination unit 14 to the server 4.

The first determination unit 12 executes the first determination based on the sensor information. The first determination unit 12 first calculates the first index value based on the captured image of the camera 21.

In the present embodiment, the first determination unit 12 calculates the drowsiness level based on the drowsiness evaluation method of NEDO (New Energy and Industrial Technology Development Organization) as the first index value. According to NEDO's drowsiness evaluation method, the drowsiness level is calculated as a level from 1 to 5. Drowsiness level 1 is not sleepy at all, sleepiness level 2 is slightly sleepy, sleepiness level 3 is sleepy, sleepiness level 4 is quite sleepy, sleepiness level 5 is very sleepy, or is beginning to sleep. Indicates the state of being.

Next, the first determination unit 12 determines whether or not the calculated first index value is equal to or greater than the first threshold value. When the first index value is equal to or greater than the first threshold value, the first determination unit 12 determines that the occupant's state is a specific state. On the other hand, when the first index value is less than the first threshold value, the first determination unit 12 determines that the state of the occupant is not a specific state.

In the present embodiment, as an example, the first determination unit 12 sets the first threshold value of the drowsiness level to 3. Then, when the calculated drowsiness level is 3 or more, it is determined that the occupant is drowsy, and when it is less than 3, it is determined that the occupant is not drowsy. The drowsiness level threshold may be appropriately set depending on how much drowsiness is allowed as "no drowsiness".

The first determination unit 12 notifies the notification unit 15 of the result of the state determination. In the case of the present embodiment, the first determination unit 12 notifies the notification unit 15 whether it is determined that the occupant is drowsy or the occupant is not drowsy.

The second determination unit 14 executes the second determination using the measurement data of the pulse sensor 22. The second determination unit 14 includes a second index value calculation unit (index value calculation unit) 141, a threshold value calculation unit (state threshold value calculation unit) 142, and a comparison determination unit (state determination unit) 143.

The second index value calculation unit 141 calculates the second index value (state index value) for determining the state based on the measurement data of the pulse sensor 22. The second index value calculation unit 141 outputs the calculated second index value to the comparison determination unit 143.

If the index value can be calculated from any of the information in the sensor group 2 and can be used to determine the state of the occupant, the type, calculation method, and value of the second index value can be determined. The unit is not particularly limited. For example, the second index value calculation unit 141 may calculate RRI (pulse interval time, RR Interval) from the measurement data of the pulse sensor 22, and calculate the second index value based on the RRI.

For example, the second index value calculation unit 141 may calculate the RRI difference as the second index value. The "RRI difference" generally indicates the value of the difference between the value of a certain RRI and the value of the RRI immediately before the certain RRI in a time series. It can be said that the RRI difference also increases because the RRI increases when the tension is relaxed. Therefore, it can be said that the RRI difference becomes large even when drowsiness becomes stronger.

In the present embodiment, the second index value calculation unit 141 calculates the second index value by the following method. The second index value calculation unit 141 calculates RRI from the measurement data of the pulse sensor 22. The calculation of RRI is continuously performed while the acquisition of measurement data continues. In parallel with the calculation of RRI, the second index value calculation unit 141 divides the measurement data into unit times and calculates the average value of RRI for each division at any time. Hereinafter, the average value of RRI per unit time is referred to as "average RRI". The unit time referred to here may be, for example, about 3 minutes.

Then, the second index value calculation unit 141 calculates the RRI difference by obtaining the difference between the latest average RRI and the previous average RRI. Therefore, the “RRI difference” in the present embodiment indicates the difference between the latest average RRI calculated by the second index value calculation unit 141 and the average RRI immediately before the average RRI.

The second index value calculation unit 141 continuously calculates the RRI difference at predetermined time intervals while the supply of sensor information from the data acquisition unit 11 continues. Then, the second index value calculation unit 141 outputs the calculated RRI difference to the sequential comparison determination unit 143. The time interval for calculating the RRI difference by the second index value calculation unit 141 is not particularly limited, but it is desirable that the time interval is such that the second determination unit 14 can specify the presence or absence of drowsiness of the occupant without delay.

The threshold value calculation unit 142 calculates a second threshold value (state threshold value) as a determination criterion in the second determination from the status log of the collected data received from the server 4. The threshold value calculation unit 142 outputs the calculated second threshold value to the comparison determination unit 143. The time interval in which the second index value calculation unit 141 calculates the second index value and the time interval in which the threshold value calculation unit 142 calculates the second threshold value may be the same or different. ..

More specifically, the threshold value calculation unit 142 first refers to the user information 31 and extracts the state log used for calculating the second threshold value from the state logs included in the collected data received by the second determination unit 14. .. If the storage device 3 does not store the user information 31, the status log does not include the attribute information of the occupant, or the status determination device 1A does not identify the individual occupant, the threshold value calculation unit 142 is the status log. You do not have to perform the extraction. That is, the threshold value calculation unit 142 may use all the state logs included in the collected data for the calculation of the second threshold value.

Subsequently, the threshold value calculation unit 142 aggregates the second index value (that is, the second index value of another person) of each of the extracted state logs, and calculates the median value of the second index value of the other person. Further, the threshold value calculation unit 142 refers to the determination result of each of the extracted status logs, and the ratio of the logs determined to be drowsy among the extracted status logs is half (50%) or more of the second index value. Specify the range. Finally, the threshold value calculation unit 142 determines the minimum second index value, which is equal to or higher than the calculated median value and the ratio of the state logs determined to be drowsy is 50% or higher, as the second threshold value. .. The threshold value calculation unit 142 calculates the mode value instead of the median value of the second index value of another person, and the ratio of the state logs determined to be the mode value or higher and drowsiness is 50% or higher. The minimum second index value may be determined as the second threshold value.

(Calculation method of threshold value)
FIG. 2 is a graph schematically showing a method of calculating the second threshold value in the threshold value calculation unit 142. The graph of FIG. 2 is for visually explaining the method of calculating the second threshold value in the threshold value calculation unit 142, and it is not necessary for the threshold value calculation unit 142 to actually create the graph shown in the figure.

The x-axis of the graph shows the second index value, that is, the RRI difference. The vertical axis of the graph shows the number of state logs showing each RRI difference (that is, the distribution of the second index value included in the collected data). Further, in the example of FIG. 2, the bar graph displayed in white shows the distribution of the RRI difference in the extracted state log, and the bar graph displayed in gray shows "sleepiness" in the second judgment of the extracted state log. The distribution of the RRI difference in the status log determined to be "yes" is shown.

In the illustrated example, the median RRI difference indicated by the extracted state log is 0.100. The ratio of the state logs whose determination result is "drowsiness" is 50% or more of the total extracted state logs when the RRI difference is 0.125 or more. In other words, the total area of the gray bar graph exceeds 50% of the total area of the white bar graph when the RRI difference is 0.125 or more. Therefore, the threshold value calculation unit 142 determines that the second threshold value is 0.125.

The comparison determination unit 143 determines the state of the occupant from the magnitude relationship between the second index value and the second threshold value. The comparison determination unit 143 determines whether or not the input second index value is equal to or greater than the input second threshold value. When the second index value is equal to or greater than the second threshold value, the comparison determination unit 143 determines that the occupant is in a specific state. On the other hand, when the second index value is equal to or less than the second threshold value, the comparison determination unit 143 determines that the occupant's condition is normal. The comparison determination unit 143 outputs the determination result to the notification unit 15.

When the first determination unit 12 determines that the occupant is in a specific state, the notification unit 15 causes the output device 5 to output the first notification. Further, the notification unit 15 causes the output device 5 to output the second notification when the second determination unit 14 (more specifically, the comparison determination unit 143) determines that the occupant is in a specific state. The contents of the first notification and the second notification are not particularly limited. For example, the first notification and the second notification may be warnings to the occupants.

(Output device 5)
The output device 5 outputs various notifications in the form of audio, video, or the like based on the control instruction of the notification unit 15. The output device 5 is realized by, for example, a speaker or a display. In addition, a plurality of output devices 5 may be used. For example, the mobile body 6 may include two or more speakers as the output device 5. Further, for example, the mobile body 6 may include both a speaker and a display as the output device 5.

(Storage device 3)
The storage device 3 stores data necessary for the operation of the state determination device 1A. For example, the storage device 3 stores the user information 31 and the status log 32. In this embodiment, the user information 31 is not essential information.

The user information 31 indicates the personal information of the occupant of the mobile body 6. The user information 31 may be, for example, information such as the gender, age, and residential area of the occupant.

FIG. 3 is a diagram showing the data structure of the state log 32. The state log 32 is log data showing a history of the second index value calculated by the second index value calculation unit 141 and the determination result of the second index value in the second determination. Further, the state log 32 may have identification information for uniquely identifying the occupant.

In the example of FIG. 3, the status log 32 has a “time” column, an “identification information” column, an “RRI difference” column, and a “determination result” column. One record of the state log 32 in FIG. 3 shows a second index value and a second determination result for a certain occupant at a certain time.

In the "time" column, the time when the second determination unit 14 generated each record is stored. The time may be measured by a timer (not shown) built in the state determination device 1A or the like. Further, the calculated time of the second index value or the time when the second determination is made may be stored in the "time" column.

In the "identification information" column, the identification information of the occupant (for example, the occupant ID) is stored. The method of identifying the individual occupant is not particularly limited. For example, the state determination device 1A may identify the occupant by analyzing the captured image of the camera 21. Alternatively, the state determination device 1A may have the occupant himself / herself input the identification information of the occupant via an input device (not shown), and identify the individual occupant from the identification information.

The RRI difference (that is, the second index value) calculated by the second index value calculation unit 141 is stored in the "RRI difference" column. The result of the second determination made by the comparison determination unit 143 is stored in the "determination result" column. In the example of FIG. 3, the record corresponding to the RRI difference determined by the comparison determination unit 143 as "no drowsiness in the occupant" is "no drowsiness", and the record corresponding to the RRI difference determined as "the occupant is drowsy" is the record. The information "drowsiness" is stored.

As described above, the second threshold value, which is the reference for the second determination, is calculated by the threshold value calculation unit 142 each time. Therefore, even if the RRI difference is the same value as in the record on the third row and the record on the fourth row in FIG. 3, the result of the second determination may be different.

(Server 4)
The server 4 collects and manages status logs from a plurality of status determination devices 1A. The server 4 includes a server communication unit 41, a server control unit 42, and a server storage unit 43.

The server communication unit 41 communicates between the server 4 and the state determination device 1A. The server storage unit 43 stores various data necessary for the operation of the server 4. The server storage unit 43 stores the collected data 431. The collected data 431 is data that summarizes the state logs collected from the state determination devices 1A of the plurality of mobile bodies 6.

The server control unit 42 may delete the record of the status log included in the collected data 431 for which a predetermined period has passed. For example, a record in which one hour has passed from the time indicated by each record may be deleted.

As a result, the capacity of the collected data 431 can be reduced. Further, as a result, the collected data 431 always includes only the status log in the latest predetermined period. The specific length of the "most recent predetermined period" is not particularly limited, but may be, for example, about one hour.

As a result, when calculating the second threshold value, the second threshold value can always be calculated based on the state logs collected within a predetermined period retroactive from the current time. Therefore, by appropriately setting a predetermined period, even if the environmental conditions that change with time such as weather and temperature change, the state log generated by the other mobile body 6 can be displayed under almost the same conditions. It can be used to calculate the second threshold. Therefore, it is possible to reduce the possibility that an erroneous determination will occur due to a difference in environmental conditions in the second determination. Therefore, the accuracy of the second determination is improved.

The server control unit 42 comprehensively manages the server 4. For example, the server control unit 42 receives a status log record from the status determination device 1A via the server communication unit 41. The server storage unit 43 adds the received status log for one record to the collected data 431. As shown in FIG. 1, the server 4 similarly collects status log records in real time from the status determination devices 1A of a plurality of mobile bodies 6. As a result, the collection data 431 of the server 4 collects and accumulates the calculation result of the RRI difference in each state determination device 1A and the determination result of the presence or absence of drowsiness in real time.

≪Processing flow≫
FIG. 4 is a flowchart showing the flow of processing in the state determination device 1A. In FIG. 4, the time when the measurement of various information in the sensor group 2 is started is shown as the start of the process.

When the measurement and acquisition of various information by the various sensors of the sensor group 2 is started, the data acquisition unit 11 acquires various sensor information from the sensor group 2 (S11). For example, the data acquisition unit 11 acquires a captured image from the camera 21 and pulse measurement data from the pulse sensor 22. Of these acquired data, the data acquisition unit 11 outputs the captured image of the camera 21 to the first determination unit 12 and the measurement data of the pulse sensor 22 to the second determination unit 14. When the sensor information used for the second determination (measurement data of the pulse sensor 22 in the present embodiment) is input to the second determination unit 14, the second determination unit 14 has sufficient sensor information for performing the second determination. Is determined (S12).

Here, "sufficient sensor information for making the second determination" is sensor information for a predetermined period (first period) to which the second index value calculation unit 141 can calculate the second index value. be. In the present embodiment, the RRI difference can be calculated if the second index value calculation unit 141 can acquire the sensor information for a period in which the average RRI can be calculated at least twice. Therefore, the sensor information sufficient for making the second determination is the measurement data of the pulse sensor 22 for a period of at least twice the calculated time interval of the average RRI.

When the second determination unit 14 has not acquired sufficient sensor information for performing the second determination (NO in S12), the second determination unit 14 instructs the first determination unit 12 to execute the first determination. Upon receiving the instruction, the first determination unit 12 executes the first determination based on the captured image input from the data acquisition unit 11 (S13). On the other hand, when the second determination unit 14 has already acquired sufficient sensor information for the second determination (YES in S12), the second determination unit 14 executes the second determination (S16).

For example, when the data acquisition unit 11 acquires the sensor information for the first time after the state determination device 1 is activated, the second determination unit 14 acquires only the first sensor information, so the determination in S12 is NO, and the first determination is made. The first determination by unit 12 is executed.

(First judgment)
FIG. 5 is a flowchart showing the flow of the first determination process. The first determination unit 12 first calculates the drowsiness level based on the captured image of the camera 21 (S131). Next, the first determination unit 12 determines whether or not the calculated drowsiness level is 3 or more (S132). When the drowsiness level is 3 or more (YES in S132), the first determination unit 12 determines that the occupant is drowsy (S133). On the other hand, when the drowsiness level is less than 3 (NO in S132), the first determination unit 12 determines that the occupant is not drowsy (S134).

This will be described again with reference to FIG. When the first determination is completed (S13), the first determination unit 12 notifies the notification unit 15 of the determination result.

The notification unit 15 determines whether or not to output the first notification to the output device 5 based on the determination result of the first determination unit 12 (S14). When it is determined that the occupant is drowsy in the first determination (YES in S14), the notification unit 15 causes the output device 5 to output the first notification (S15). On the other hand, when it is determined that the occupant is not drowsy (NO in S14), the notification unit 15 ends the process without causing the output device 5 to output the first notification. Next, when the data acquisition unit 11 acquires various sensor information, the state determination device 1A executes processing from S11 again. When the second determination unit 14 can acquire sufficient sensor information for the second determination (YES in S12), the second determination is executed (S16).

(Second judgment)
FIG. 6 is a flowchart showing the flow of the second determination process. The second index value calculation unit 141 of the second determination unit 14 calculates the second index value, that is, the RRI difference, based on the measurement data of the pulse sensor 22 (S161). The second index value calculation unit 141 outputs the RRI difference to the comparison determination unit 143.

On the other hand, the second determination unit 14 receives the collected data from the server 4 via the communication unit 13 (S162). The threshold value calculation unit 142 of the second determination unit 14 refers to the user information 31 and extracts the status log used for calculating the second threshold value from the status logs included in the collected data received by the second determination unit 14. (S163). The process of S163 is not essential. When the status log extraction is not executed, the threshold value calculation unit 142 may use all the status logs included in the collected data in the subsequent processing.

Subsequently, the threshold value calculation unit 142 aggregates the second index value (that is, the second index value of another person) of each of the extracted state logs, and calculates the median value of the second index value of the other person (S164). Further, the threshold value calculation unit 142 refers to the determination result of each of the extracted state logs, and specifies the value of the second index value in which the ratio of the logs determined to be drowsy is 50% or more among the extracted state logs. (S165). Finally, the threshold value calculation unit 142 determines the minimum second index value that is equal to or higher than the calculated median and the proportion of logs determined to be drowsy is 50% or higher as the second threshold value (the second threshold value). S166). The threshold value calculation unit 142 outputs the determined second threshold value to the comparison determination unit 143.

The comparison determination unit 143 determines whether or not the RRI difference calculated by the second index value calculation unit 141 is equal to or greater than the second threshold value (S167). When the RRI difference is equal to or greater than the second threshold value (YES in S167), the comparison determination unit 143 determines that the occupant is drowsy (S168). On the other hand, when the RRI difference is less than the second threshold value (NO in S167), the comparison determination unit 143 determines that the occupant is not drowsy (S169).

This will be described again with reference to FIG. When the second determination is completed (S16), the comparison determination unit 143 notifies the notification unit 15 of the determination result. The notification unit 15 determines whether or not to output the second notification to the output device 5 based on the determination result of the comparison determination unit 143 (S17). When it is determined that the occupant is drowsy in the second determination (YES in S17), the notification unit 15 causes the output device 5 to output the second notification (S18). On the other hand, when it is determined that the occupant is not drowsy (NO in S17), the notification unit 15 ends the process without causing the output device 5 to output the second notification.

Next, when the data acquisition unit 11 acquires various sensor information, the state determination device 1A executes processing from S11 again. Since the measurement data of the pulse sensor 22 is sequentially supplied to the data acquisition unit 11, the second determination unit 14 is in a state where sufficient sensor information for the second determination is accumulated thereafter. Therefore, the second determination unit 14 repeatedly executes the second determination (S16) every time the sensor information is acquired from the data acquisition unit 11.

After performing the second determination (S16), the second determination unit 14 may create a status log and store it in the storage device 3 as one record of the status log 32. Further, the second determination unit 14 may transmit the created status log (that is, the status log for one record) to the server 4 via the communication unit 13. The server storage unit 43 of the server 4 receives the status log for one record via the server communication unit 41. The server storage unit 43 adds the received status log for one record to the collected data 431.

When the first determination is not executed, the state determination device 1A does not execute the processes of S13 to S15 in FIG. 4 until sufficient measurement data for performing the second determination is obtained (YES in S12). You may be waiting. Alternatively, the second determination may be executed even when the sensor information for a sufficient period for performing the second determination has not been obtained (S16). For example, if the sensor information for a period sufficient for performing the second determination has not been obtained, the second determination unit 14 may consider that the second index value is 0 and execute the second determination. ..

(Modification example)
In the present embodiment, the types of the first index value and the second index value and the method of calculating the values may be appropriately determined according to the data that can be acquired from the sensor group 2. That is, the first determination unit 12 and the second determination unit 14 may each calculate an index value that can be calculated from the information obtained from the sensor group 2 as a first index value or a second index value. Then, the data acquisition unit 11 may supply the information required by the first determination unit 12 and the second determination unit 14 to the first determination unit 12 and the second determination unit 14 as appropriate.

Further, for example, the threshold value calculation unit 142 of the second determination unit 14 may calculate the following values as the second index value. That is, the threshold value calculation unit 142 may calculate the pulse rate per unit time as the second index value. Further, the threshold value calculation unit 142 may calculate at least one of the maximum change width of RRI per unit time, the average value of RRI per unit time, and the median value of RRI per unit time as the second index value. .. Further, the threshold value calculation unit 142 may calculate any of the integrated values of the high frequency component, the low frequency component, the ultra low frequency component, and the ultra low frequency component of the RRI per unit time as the second index value. Further, the threshold value calculation unit 142 may calculate the ratio of the integrated values of the frequency components having different RRIs as the second index value, such as the ratio of the integrated values of the high frequency components and the low frequency components of the RRI. Further, the threshold value calculation unit 142 may calculate the RRI simple regression coefficient as the second index value.

Further, for example, the second determination unit 14 may analyze the captured image of the camera 21 and calculate the second index value based on the analysis result. For example, the threshold value calculation unit 142 may specify the movement of the occupant's eyelids from the captured image. Specifically, the threshold value calculation unit 142 may calculate the eye closing time, the number of times the eye is closed or blinking per unit time, and the like as the second index value.

[Embodiment 2]
The state determination system 100 may consider the past second index value of the occupant when determining the state of the occupant. Hereinafter, the second embodiment of the present disclosure will be described. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the first embodiment, and the description will not be repeated.

≪Main part composition≫
FIG. 7 is a block diagram showing a configuration of a main part of the state determination system 200 according to the present embodiment. The state determination system 200 differs from the state determination system 100 in that the state determination device 1B is included. The state determination device 1B differs from the state determination device 1A in that it has a third determination unit (average determination unit) 16.

When the second index value calculation unit 141 of the second determination unit 14 according to the present embodiment calculates the second index value, the second index value is output to the third determination unit 16. When the third determination unit 16 determines that the occupant is in a specific state, the notification unit 15 according to the present embodiment causes the output device 5 to output the third notification. The content of the third notification may be a warning to the occupants as in the first notification and the second notification.

The third determination unit 16 executes the third determination based on the second index value. The third determination is to compare the average value of the past second index values of the occupants of the moving body 6 with the latest second index value obtained from the second index value calculation unit 141. Is a determination method for determining whether or not is in a specific state. The details of the third determination will be described later. The third determination unit 16 notifies the notification unit 15 of the determination result.

≪Processing flow≫
FIG. 8 is a flowchart showing a processing flow in the state determination device 1B according to the present embodiment. Note that the parts with the same step numbers as in FIG. 4 in FIG. 8 are the same processes as in FIG. 4, and therefore the description will not be repeated.

The state determination device 1B executes the third determination in parallel with the second determination (S19). As described above, the second index value is used for the third determination. Therefore, the third determination unit 16 acquires the second index value at least once from the second index value calculation unit 141, and then executes the third determination. Therefore, when the second determination is made for the first time, the third determination does not have to be executed in parallel.

FIG. 9 is a flowchart showing the flow of the process of the third determination. When the third determination unit 16 acquires the second index value, that is, the RRI difference (S191), the third determination unit 16 reads out the occupant status log for a predetermined second period. The "second period" referred to here is, for example, about 10 hours retroactively from the current time. Next, the third determination unit 16 calculates the average value of the RRI differences indicated by each record of the read state log (S192). Subsequently, the third determination unit 16 has the RRI difference calculated by the second index value calculation unit 141 (that is, the current RRI difference of the occupant) and the average value of the RRI difference of the status log for the second period + 3σ. Compare the magnitude (S193). Note that "3σ" is a value that defines an allowable range of error from the average value.

When the RRI difference calculated by the second index value calculation unit 141 is equal to or greater than the average value of the RRI difference of the status log for the second period + 3σ (YES in S193), the third determination unit 16 is drowsy to the occupant. (S194). When the RRI difference calculated by the second index value calculation unit 141 is less than the average value + 3σ of the RRI difference of the status log for the second period (NO in S193), the third determination unit 16 does not make the occupant drowsy. (S195).

It is not essential to add an error of 3σ to the average value. That is, the third determination unit 16 may compare and determine the RRI difference and the average value instead of the RRI difference and the average value + 3σ calculated by the second index value calculation unit 141 in S193. Then, if the RRI difference is greater than or equal to the average value, it may be determined that the occupant is drowsy, and if it is less than the average value, it may be determined that the occupant is not drowsy.

The explanation will be given by returning to FIG. 8 again. When the third determination is completed (S19), the third determination unit 16 notifies the notification unit 15 of the determination result. The notification unit 15 determines whether or not to output the third notification to the output device 5 based on the determination result of the third determination unit 16 (S20). When it is determined that the occupant is drowsy in the third determination (YES in S20), the notification unit 15 causes the output device 5 to output the third notification (S21). On the other hand, when it is determined that the occupant is not drowsy (NO in S20), the notification unit 15 ends the process without causing the output device 5 to output the third notification.

If the currently calculated second index value is larger than the average value of the occupant's own second index value, that is, the value estimated to be the occupant's second index value at normal times, the occupant's condition is normal. Most likely different. According to the above configuration, the state determination device can determine that the occupant has reached a specific state when the occupant's state is likely to be different from the normal state.

≪RRI difference appearance frequency and 3rd judgment≫
FIG. 10 is a graph showing the appearance frequency of the RRI difference of a certain occupant in the second period. The graph in FIG. 10 is for visually explaining the determination process in the third determination unit 16, and it is not necessary for the third determination unit 16 to actually create the illustrated graph.

The frequency of occurrence of individual RRI differences usually converges in a form similar to a normal distribution, as shown in FIG. As shown in FIG. 10, the value farther from the average value in the graph, the lower the frequency of appearance. Therefore, it can be said that the farther the RRI difference of the current occupant is from the average value shown in FIG. 10, the higher the possibility that the occupant is in a different state from the normal state.

Therefore, by adjusting the value of the error 3σ, it is possible to adjust the threshold value of the appearance frequency of whether the RRI difference of a certain occupant is determined to be “normal” or “abnormal”. For example, in FIG. 10, it is assumed that the average value of the RRI difference + 3σ is set to a value at which the appearance frequency shown on the vertical axis is 5%. In this assumption, when the current RRI difference of the occupant calculated by the second index value calculation unit 141 is equal to or more than the mean value of the RRI difference + 3σ, the appearance frequency of the current RRI difference is 5% or less (graph in FIG. 10). The value in the shaded area of). Therefore, it can be determined that the current RRI difference for the occupant is an abnormal value that is usually difficult to calculate.

In this way, the third determination unit 16 determines whether or not the occupant is drowsy by performing the third determination for determining whether or not the RRI difference shows an abnormal value for the individual occupant. According to this third determination, it is possible to adjust how far the RRI difference is from the average value to determine that it is abnormal, depending on the magnitude of the value of "3σ" to be added to the average value.

The third determination unit 16 according to the present embodiment may count the number of times that the second index value calculated by the second index value calculation unit 141 shows an abnormal value. Then, when the number of times that the second index value shows an abnormal value is less than the predetermined number of times, the third determination unit 16 determines that the occupant is not in a specific state, and the number of times that the abnormal value is shown is the predetermined number of times. In the above cases, it may be determined that the occupant is in a specific state. The count of this number of times may be reset to 0 if the third determination unit 16 makes a determination once.

The specific number of "predetermined times" is not limited. For example, the predetermined number of times may be 10 times. In this way, when an abnormal value is detected more than a certain number of times, it is determined that the occupant has entered a specific state, so that the value temporarily rises significantly. It is possible to prevent making a judgment. Therefore, the accuracy of the third determination can be improved.

The third determination unit 16 may set a time limit until the count is reset when counting the above-mentioned "number of times when an abnormal value is shown". That is, when the third determination unit 16 has a predetermined number of times or more and the second index value is an average value + 3σ or more within a predetermined time, the third determination unit 16 may determine that the occupant's condition is abnormal. Then, when the predetermined time has elapsed, the count of the number of times when the second index value is the average value + 3σ may be reset. Thereby, the accuracy of the third determination can be further improved.

The third determination unit 16 according to the present embodiment may use the value of the intermediate result when calculating the second index value from the sensor information, instead of the second index value of the occupant, for the third determination. When the value of the intermediate result is used for the third determination, the second determination unit 14 includes the history of the value of the intermediate result in place of the second index value or together with the second index value, the state log 32. To create.

Specifically, for example, the third determination unit 16 may execute the third determination using the average RRI instead of the RRI difference. When the third determination is executed using the average RRI, the second determination unit 14 creates a state log 32 including the history of the average RRI and stores it in the storage device 3. Further, when the third determination is executed using the average RRI, the second index value calculation unit 141 outputs the average RRI to the third determination unit 16. Subsequent processing may be performed by replacing the "RRI difference" in the processing described in the present embodiment with the "average RRI".

[Embodiment 3]
The state determination system 300 according to the present embodiment is a system for determining the state of an occupant staying in the internal space of the moving body, like the state determination systems 100 and 200. There are various methods for determining whether or not a person is in a specific state such as drowsiness by using an index value indicating the state. However, an effective determination method, that is, an index value having a high correlation with the above-mentioned "specific state", differs from person to person.

The state determination system 300 according to the present embodiment performs state determination in consideration of the above-mentioned individual differences. Specifically, the state determination system 300 calculates a plurality of types of index values for specifying the state of the occupant by a plurality of methods. Next, the state determination system 300 selects an index value having a high correlation among the individual occupants, and calculates a score indicating the possibility that the occupant is in a specific state according to the value of the index value. Then, the state determination system 300 determines whether or not the total of the calculated scores is equal to or greater than a predetermined third threshold value (score threshold value). When the total score is equal to or greater than the third threshold value, the state determination system 300 makes a final determination result that the occupant is in a specific state.

Hereinafter, the third embodiment of the present disclosure will be described in detail with reference to FIGS. 11 to 16. The same reference numerals are given to the members having the same functions as the members described in the first and second embodiments, and the description is not repeated.

≪Main part composition≫
FIG. 11 is a block diagram showing a configuration of a main part of the state determination system 300 according to the present embodiment. Unlike the state determination systems 100 and 200, the state determination system 300 does not have to include the server 4.

The state determination system 300 includes a state determination device 1C. Further, the storage device 3 according to the state determination system 300 stores the user information 31, the state log 32, and the explanatory variable data 33. In this embodiment, the user information 31 and the status log 32 are not essential data.

(Status determination device 1C)
The state determination device 1C does not have to have the communication unit 13. Further, the state determination device 1C is not a second determination unit 14, but has a plurality of explanatory variable calculation units 17 and a plurality of score calculation units 18 corresponding to the explanatory variable calculation unit 17. Different from 1A and 1B. Although not included in the example of FIG. 11, the state determination device 1C may include the first determination unit 12. In the example of FIG. 11, the state determination device 1C includes three explanatory variable calculation units 17A to 17C as an example of the plurality of explanatory variable calculation units 17. Further, in the example of FIG. 11, the state determination device 1C includes score calculation units 18A to 18C corresponding to the explanatory variable calculation units 17A to 17C as an example of the plurality of score calculation units 18. Hereinafter, when the explanatory variable calculation units 17A to 17C are collectively referred to, they are also referred to as the explanatory variable calculation units 17. Further, when the score calculation units 18A to 18C are generically referred to hereinafter, they are also referred to as the score calculation unit 18. Further, the state determination device 1C includes a score determination unit 19, an identification unit 70, and an explanatory variable selection unit 71. The data acquisition unit 11 according to the present embodiment outputs sensor information to the identification unit 70 and the explanatory variable calculation unit 17, respectively.

The identification unit 70 acquires the sensor information of the sensor group 2 and identifies the occupant from the sensor information. For example, the identification unit 70 may identify the occupant from the image taken by the camera 21. Further, when the sensor group 2 includes a microphone, the identification unit 70 may identify the occupant by recognizing the utterance voice of the occupant acquired by the microphone. The identification unit 70 outputs the identification result of the occupant to the explanatory variable selection unit 71.

The explanatory variable selection unit 71 reads out the explanatory variable data 33 corresponding to the occupant identified by the identification unit 70, and selects the explanatory variable to be used for determining the state of the occupant. The details of the method of selecting the explanatory variables will be described later. The explanatory variable selection unit 71 instructs the explanatory variable calculation unit 17 corresponding to the explanatory variable used for determining the state of the occupant to calculate the explanatory variable.

Each of the explanatory variable calculation units 17A to 17C calculates an index value indicating whether or not the occupant is in a specific state by using different methods based on the sensor information obtained from the sensor group 2. That is, in the present embodiment, the explanatory variable calculation units 17A to 17C each calculate an index value indicating the presence or absence of drowsiness of the occupant. The index values calculated by the explanatory variable calculation units 17A to 17C are explanatory variables for finally obtaining the objective variable indicating "whether or not the occupant is in a specific state". Hereinafter, the index value calculated by each of the explanatory variable calculation units 17 is also referred to as an explanatory variable.

The specific calculation method for each explanatory variable is not limited. For example, any of the explanatory variable calculation units 17A to 17C may calculate the drowsiness level as an explanatory variable, as in the first determination unit 12. Further, in the state determination device 1C, the number of the explanatory variable calculation unit 17 and the score calculation unit 18 is not particularly limited as long as they are plural. The explanatory variable calculation units 17A to 17C output the calculated explanatory variables to the score calculation units 18A to 18C, respectively.

The score calculation units 18A to 18C calculate the score based on the explanatory variables input from the corresponding explanatory variable calculation units 17A to 17C, respectively. Specifically, when the explanatory variables are input, the score calculation units 18A to 18C refer to the explanatory variable data 33 and determine whether or not each explanatory variable is equal to or greater than the overall threshold value and equal to or greater than the individual threshold value. .. When the index value is equal to or higher than the overall threshold value and equal to or higher than the individual threshold value, the score calculation units 18A to 18C substitute the index value into the score formula of each index value stored in the explanatory variable data 33, respectively. Calculate the score of the index value. The score calculation units 18A to 18C output the calculated scores to the score determination unit 19, respectively. The overall threshold, individual threshold, and score formula will be described in detail later.

The score determination unit 19 calculates a total score, which is the sum of the scores, and determines whether or not the occupant is in a specific state based on the total score. In the present embodiment, it can be said that the determination result of the score determination unit 19 is the final result of the occupant's state determination.

Specifically, the score determination unit 19 determines that the occupant is in a specific state when the total score is equal to or higher than the predetermined third threshold value, and when the total score is less than the predetermined third threshold value, the occupant determines. Judge that it is not in a specific state. That is, in the present embodiment, the score determination unit 19 determines that the occupant is drowsy when the total score is equal to or higher than the third threshold value, and determines that the occupant is not drowsy when the total score is less than the third threshold value. The score determination unit 19 may normalize the value of the total score before determining the presence or absence of drowsiness. The specific value of the third threshold value may be appropriately determined according to the range of values of the explanatory variables. For example, the third threshold value may be about 0.5 when the value of each explanatory variable is in the range of 0 to 1. The score determination unit 19 outputs the determination result to the notification unit 15.

When the score determination unit 19 determines that the occupant is in a specific state, the notification unit 15 according to the present embodiment causes the output device 5 to output a fourth notification. The content of the fourth notification may be, for example, a warning to the occupant as in the first notification to the third notification.

(Explanatory variable data 33)
The explanatory variable data 33 is data in which the overall threshold value and the individual threshold value, which are the determination criteria for each explanatory variable, are recorded. Further, the explanatory variable data 33 is data indicating whether or not each occupant is in a specific state and the strength of the correlation with each explanatory variable. The explanatory variable data 33 is created in advance for each occupant using the state determination device 1C or another device, and is stored in the storage device 3.

FIG. 12 is a diagram showing a data structure of the explanatory variable data 33. The explanatory variable data 33 is data in which the explanatory variable name is associated with the overall threshold value, the individual threshold value, and the score expression of the explanatory variable. One record of the explanatory variable data 33 indicates explanatory variable data for one explanatory variable. In FIG. 12, the "explanatory variable" column stores information indicating the explanatory variable name. The overall threshold is stored in the "overall threshold" column. The individual threshold is stored in the "individual threshold" column. Information indicating the score expression is stored in the "score expression". In the example of FIG. 12, the explanatory variable data 33 further includes information indicating the priority of each explanatory variable.

(Method of creating explanatory variable data 33)
The explanatory variable data 33 is created based on sample data of an individual occupant (hereinafter referred to as personal data) and sample data of a plurality of persons (hereinafter referred to as overall data). In addition, personal data may be included in the whole data. Further, it is preferable that the number of people constituting the entire data is large.

Both the personal data and the whole data have the value of the explanatory variable (that is, the index value) calculated by the same method as that of the explanatory variable calculation unit 17, and the probability that the index value is determined to be "sleepy". It is the data associated with. The probability of being determined to be "sleepy" is determined by measuring whether or not the occupant was actually in a specific state in each index value included in the overall data or personal data, and using the measured data by a method such as logistic regression. It can be calculated by classifying the values. For example, the personal data and the overall data may be data in which the drowsiness level described in the first embodiment and the probability that the subject actually causes drowsiness at the drowsiness level are associated with each other.

FIG. 13 is a graph showing global thresholds, individual thresholds, and score equations when there is no correlation between an explanatory variable and the change of occupant to a particular state. FIG. 14 is a graph showing global thresholds, individual thresholds, and score equations when there is a correlation between the same explanatory variables as in FIG. 13 and changes to a particular state of the occupant. The horizontal axis of FIGS. 13 and 14 shows the index value.

The vertical axis of FIGS. 13 and 14 indicates the probability of being determined to be “sleepy” in the index value indicated by the horizontal axis. The solid bar graphs in FIGS. 13 and 14 show the distribution of personal data. On the other hand, the dotted bar graph in FIG. 13 shows the distribution of the entire data. The graphs of FIGS. 13 and 14 are for visually explaining the calculation method of each value in the explanatory variable data 33, and it is not necessary for the state determination device 1C to actually create the illustrated graph.

The "overall threshold" is an average value of the probabilities that the occupant is determined to be in a specific state (in this embodiment, drowsiness) in all the samples of the overall data. The "individual threshold" is an average value of the probabilities that the occupant is determined to be in a specific state in all samples of personal data. The "score formula" is a formula showing the correlation between the explanatory variables and the probability that the occupant is in a specific state. As shown in the figure, the score expression is, for example, a variable expression of a smoothing curve obtained by smoothing personal data. As will be described later, the value obtained by substituting the value of the explanatory variable into the score expression becomes the score of the explanatory variable. That is, the "score" in the present embodiment is a numerical value calculated for each explanatory variable and is a value indicating the possibility that the occupant is in a specific state.

As shown in FIG. 13, the value of the explanatory variable of personal data when there is no correlation in the change of a certain occupant to a specific state even if the explanatory variable generally has a correlation with a specific state. Even if the value fluctuates, the probability of being determined to be "sleepy" does not fluctuate significantly. On the other hand, if the value of the explanatory variable of the entire data fluctuates, the probability of being determined to be "sleepy" increases. Therefore, as shown in FIG. 13, the individual threshold is lower than the overall threshold.

On the other hand, as shown in FIG. 14, in the case of an explanatory variable having a correlation with the change of a certain occupant to a specific state, if the value of the explanatory variable of personal data fluctuates, the probability of being determined to be "sleepy" increases. do. When the explanatory variables of the personal data have a higher degree of correlation than the explanatory variables of the overall data, the individual threshold value is higher than the overall threshold value, as shown in FIG.

In this way, the state determination device 1C or another device calculates in advance the overall threshold value, the individual threshold value for each occupant, and the score expression for each explanatory variable, and records them in the explanatory variable data 33. As a result, the explanatory variable calculation unit 17 (described later) of the state determination device 1C can calculate the score from the calculated explanatory variable values.

≪Processing flow≫
FIG. 15 is a flowchart showing the flow of processing in the state determination device 1C. In FIG. 15, the time when the measurement of various information in the sensor group 2 is started is shown as the start of the process. When the image capture by the camera 21 and the pulse measurement by the pulse sensor 22 are started (S21), the data acquisition unit 11 continuously acquires various sensor information from the sensor group 2. The data acquisition unit 11 outputs the acquired sensor information to the identification unit 70 and the explanatory variable calculation units 17A to 17C, respectively. The data acquisition unit 11 does not need to output all the sensor information to all the explanatory variable calculation units 17. At least, the data acquisition unit 11 may output the sensor information necessary for calculating the explanatory variables in each of the explanatory variable calculation units 17A to 17C to each of the explanatory variable calculation units 17A to 17C.

The identification unit 70 identifies the occupant based on the sensor information, and outputs the identification result to the explanatory variable selection unit 71. The explanatory variable selection unit 71 reads out the explanatory variable data 33 corresponding to the occupant identified by the identification unit 70, and selects the explanatory variable to be used for determining the state of the occupant (S22). The explanatory variable selection unit 71 instructs the explanatory variable calculation unit 17 corresponding to the explanatory variable used for determining the state of the occupant to calculate the explanatory variable.

Hereinafter, each explanatory variable is calculated and the score is calculated by each explanatory variable calculation unit 17 instructed by the explanatory variable selection unit 71 and the score calculation unit 18 corresponding to these. As long as it is S22 or later, the order and timing of calculation of each explanatory variable and score calculation are not particularly limited.

First, each explanatory variable calculation unit 17 calculates an index value to be an explanatory variable (S23). The explanatory variable calculation unit 17 outputs the calculated explanatory variables to the corresponding score calculation unit 18. When the explanatory variables are input, the score calculation unit 18 refers to the explanatory variable data 33 to specify the overall threshold value and the individual threshold value of the input explanatory variables. Each of the score calculation units 18 determines whether or not the explanatory variables are equal to or greater than the overall threshold value corresponding to the explanatory variables and equal to or greater than the individual threshold value (S24).

When the explanatory variable is equal to or greater than the overall threshold value and equal to or greater than the individual threshold value (YES in S24), the score calculation unit 18 that has made the determination determines the score according to the value of the explanatory variable (S25). For example, the score calculation unit 18 which was YES in S23 obtains a score by substituting the value of the explanatory variable into the score formula of the explanatory variable stored in the explanatory variable data 33.

On the other hand, when the explanatory variable is less than the overall threshold value or at least one of the explanatory variables is less than the individual threshold value (NO in S24), the score calculation unit 18 that makes the determination sets the score of the explanatory variable to 0 (NO). S26).

In either case, each score calculation unit 18 outputs the calculated score to the score determination unit 19. When the calculation of all explanatory variables and the score calculation of all explanatory variables are completed, the score determination unit 19 calculates the total score by summing the input scores (S27). The score determination unit 19 further determines whether or not the total score is equal to or greater than the third threshold value (S28). When the total score is equal to or higher than the third threshold value (YES in S28), the score determination unit 19 determines that the occupant is drowsy (S29). Upon receiving the determination result, the notification unit 15 causes the output device 5 to output a fourth notification (S30). On the other hand, when the total score is less than the third threshold value (NO in S28), the score determination unit 19 determines that the occupant is not drowsy (S31). In this case, the notification unit 15 does not output the fourth notification to the output device 5, and ends the process.

≪Selection of explanatory variables≫
Finally, the selection of the explanatory variables by the explanatory variable selection unit 71 will be described. FIG. 16 is a diagram schematically showing the selection of an explanatory variable used for determining the state of a certain occupant and the result of the state determination using the explanatory variable. For convenience, in the example of FIG. 16, the occupant identified by the identification unit 70 is referred to as an occupant A. Further, in FIG. 16, the third threshold value is 0.5.

The first line of FIG. 16 shows the types of explanatory variables. In the example of FIG. 16, it is assumed that the state determination device 1C has six explanatory variable calculation units 17 and six score calculation units 18 corresponding to the six types of explanatory variables X1 to X6. The explanatory variable selection unit 71 selects the explanatory variable to be used for determining the state of the occupant A by referring to the explanatory variable data 33 of the occupant A. In the illustrated example, the explanatory variable selection unit 71 selects four explanatory variables X1 to X4 as explanatory variables to be used for determining the state of the occupant, in order from the top of the priority stored in the explanatory variable data 33. ..

The number of explanatory variables to be selected is not particularly limited. Further, the method of selecting the explanatory variables is not limited to the above method. For example, the explanatory variable selection unit 71 may use a predetermined number of explanatory variables in order of increasing individual threshold value in the explanatory variable data 33 of the occupant A for determining the state of the occupant. Alternatively, the state determination device 1C refers to the score expression of each explanatory variable in the explanatory variable data 33 of the occupant A, and uses a predetermined number of explanatory variables in order of increasing maximum value of the score expression to determine the state of the occupant. It may be used.

The second line of FIG. 16 shows the comparison result between the value of each explanatory variable and the overall threshold value and the individual threshold value. Further, the third line of FIG. 16 shows the score of each explanatory variable. In addition, in FIG. 16, the “threshold value” indicates a threshold value having a larger value among the overall threshold value and the individual threshold value. That is, when the value of the explanatory variable is equal to or greater than the threshold value in FIG. 16, it means that the value of the explanatory variable is equal to or greater than the overall threshold value and equal to or greater than the individual threshold value. In the illustrated example, since the values of the explanatory variables X1 to X3 are equal to or greater than the threshold value, each score calculation unit 18 substitutes the values of the explanatory variables X1 to X3 into the score expressions of the explanatory variables. Then, find the score. On the other hand, the value of the explanatory variable X4 is equal to or less than the threshold value. Therefore, the score calculation unit 18 corresponding to the explanatory variable X4 sets the score of the explanatory variable X4 to 0.

Thereby, when it is estimated from the value of the explanatory variable that the occupant A is unlikely to feel drowsy, the score of the explanatory variable can be set to 0 so as not to contribute to the total score. Therefore, the accuracy of the state determination can be improved.

In the case of FIG. 16, as shown in the figure, the score determination unit 19 totals the scores derived from the explanatory variables X1, X2, and X3 to calculate the total score. Then, since the total score is 0.5 or more, which is the third threshold value, it is determined that the occupant A is drowsy.

According to the process according to the present embodiment, it is possible to select explanatory variables that are highly correlated with the change of the occupant to a specific state, and sum the scores of the explanatory variables to obtain the final state judgment result. .. Therefore, the state of the occupant can be determined more accurately.

In other words, according to the process according to the present embodiment, it is possible to prevent the explanatory variables having a low correlation with the change of the occupant to a specific state from being used for the state determination. Therefore, it is possible to accurately determine whether or not the occupant is in a specific state.

[Embodiment 4]
The state determination method according to the first or second embodiment and the state determination method according to the third embodiment may be used in combination. Hereinafter, a fourth embodiment of the present disclosure will be described. The same reference numerals are given to the members having the same functions as the members described in any one of the first to third embodiments, and the description is not repeated.

FIG. 17 is a block diagram showing a configuration of a main part of the state determination system 400 according to the present embodiment. The state determination system 400 has a configuration in which the state determination system 100 according to the first embodiment and the state determination system 300 according to the third embodiment are combined. Although the internal configuration of the server 4 is not shown in FIG. 17 for the sake of simplification of the drawings, the server 4 has the same configuration as the server 4 described in the first and second embodiments.

The storage device 3 according to the present embodiment stores the user information 31, the state log 32, and the explanatory variable data 33. Further, the state determination device 1D according to the present embodiment includes a plurality of second determination units 14 in place of the explanatory variable calculation unit 17 of the state determination device 1C. In the example of FIG. 17, the state determination device 1D includes three second determination units 14A to 14C.

Hereinafter, the processing procedure for determining the state of the occupant in the state determination device 1D will be described. The second determination units 14A to 14C have the same configuration as the second determination unit 14 described in the first and second embodiments. The second determination units 14A to 14C each execute the second determination described in the first embodiment. The start timing of the second determination is the same as in FIG. When the second determination units 14A to 14C determine that the occupant is in a specific state, that is, "sleepy", the corresponding score calculation units 18A to 18C are notified of the second index used for the second determination. Output the value. The second determination units 14A to 14C do not have to output the determination result to the notification unit 15. Further, the notification unit 15 does not have to output the second notification.

The score calculation units 18A to 18C each calculate the score of each explanatory variable using the input second index value as an explanatory variable, as in the third embodiment. The score calculation units 18A to 18C output the calculated scores to the score determination unit 19, respectively. Subsequent processing is the same as that of the state determination device 1C according to the third embodiment.

When the state determination device 1D according to the present embodiment performs the second determination as described above, the explanatory variable data 33 does not have to include information indicating the overall threshold value and the individual threshold value. Further, in this case, the state determination device 1D executes only the second determination related to the calculation of the second index value for a predetermined number from the higher priority in the explanatory variable data 33 among the plurality of second determinations. You may. For example, in the state determination device 1D capable of executing six types of second determinations, only the determinations of the top four priorities may be executed.

Further, the state determination device 1D according to the present embodiment may determine the state of the occupant by the following processing procedure. The second determination units 14A to 14C include at least the second index value calculation unit 141 and the comparison determination unit 143 described in the first and second embodiments. The second determination units 14A to 14C each execute the second determination described in the first embodiment. However, the second determination units 14A to 14C do not execute the determination of the second threshold value shown in S162 to S166 of FIG. 6 and the comparison determination of the second index value and the second threshold value shown in S167.

The second determination units 14A to 14C refer to the explanatory variable data 33 instead of S162 to S166 and S167 in FIG. 6 to execute the process of S24 in FIG. That is, the second determination units 14A to 14C determine that the occupant is in a specific state, that is, "sleepy" when the calculated second index value is equal to or greater than the overall threshold value and equal to or greater than the individual threshold value. do. Then, when it is determined that the state is specific, the second determination units 14A to 14C output the second index value to the corresponding score calculation units 18A to 18C as explanatory variables.

The score calculation units 18A to 18C each calculate the score of each explanatory variable using the input second index value as an explanatory variable, as in the third embodiment. The score calculation units 18A to 18C output the calculated scores to the score determination unit 19, respectively. Subsequent processing is the same as that of the state determination device 1C according to the third embodiment.

The state determination system 400 according to the present embodiment may be configured by combining the state determination system 200 according to the second embodiment and the state determination system 300 according to the third embodiment. That is, the state determination device 1D may include a third determination unit 16. The function of the third determination unit 16 is the same as that described in the second embodiment. In this case, the various processes in the second determination unit 14A to 14C, the score calculation units 18A to 18C, and the score determination unit 19 described above and the third determination in the third determination unit 16 may be performed in parallel. ..

[Modification example]
A part or all of various processes in the state determination devices 1A to 1D according to each embodiment of the present disclosure may be executed by the server 4. For example, the state determination devices 1A to 1D according to the first embodiment include a data acquisition unit 11, a communication unit 13, and a notification unit 15, and the processing performed by other members of the state determination devices 1A to 1D is The server control unit 42 of the server 4 may execute the data. In this case, the state determination devices 1A to 1D transmit the sensor information acquired by the data acquisition unit 11 to the server 4 via the communication unit 13. Further, the communication unit 13 of the state determination devices 1A to 1D receives at least one of the determination results of the first determination, the second determination, the third determination, and the status determination using the total score from the server 4. The determination result is output to the notification unit 15. As described in each of the above-described embodiments, the notification unit 15 determines whether or not to output various notifications to the output device 5 according to the determination result, and if the output is to be output, one of the first to fourth notifications. Is output to the output device 5.

Further, the state determination devices 1A to 1D according to each embodiment of the present disclosure may be devices for determining various states of the occupant regardless of the presence or absence of drowsiness of the occupant. For example, the state determination devices 1A to 1D may be devices for determining whether or not the occupant is in an inappropriate state for at least one of driving and communication. Here, the "state inappropriate for at least one of driving or communication" is, for example, an angry state, a dull state, or the like. Further, the various sensors included in the sensor group 2 may be sensors capable of acquiring information necessary for the state determination devices 1A to 1D to specify the state to be determined. For example, when the state determination devices 1A to 1D are devices for determining whether or not the occupant is in an angry state, the sensor group 2 may include the camera 21 and the pressure sensor installed on the seat. good. Then, the index value for specifying the state used for various determinations may be specified from the facial expression of the occupant in the image taken by the camera 21. Further, the index value may be specified from the movement of the occupant during sitting, which is indicated by the measurement data of the pressure sensor.

[Example of implementation by software]
The control blocks of the state determination devices 1A, 1B, 1C, 1D, and the server 4 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software. good.

In the latter case, the state determination devices 1A, 1B, 1C, 1D, and the server 4 include a computer that executes a program instruction that is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. Then, in the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present disclosure. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium", for example, a ROM (Read Only Memory) or the like, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the above program may be further provided. Further, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that one aspect of the present disclosure can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

The present disclosure is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present disclosure.

100, 200, 300, 400 Status determination system 1A, 1B, 1C, 1D Status determination device 11 Data acquisition unit 12 First determination unit 13 Communication unit 14 Second determination unit 141 Second index value calculation unit 142 Threshold calculation unit 143 Comparison Judgment unit 15 Notification unit 16 Third judgment unit 17 Explanation variable calculation unit 18 Score calculation unit 19 Score judgment unit 70 Identification unit 71 Explanation variable selection unit 2 Sensor group 21 Camera 22 Pulse sensor 3 Storage device 31 User information 32 Status log 33 Explanation Variable data 4 Server 41 Server communication unit 42 Server control unit 43 Server storage unit 431 Collected data 5 Output device 6 Mobile

Claims (10)

An acquisition unit that acquires sensor information about the occupant of the moving body from one or more sensor devices arranged in the internal space of the moving body, and an acquisition unit.
An explanatory variable selection unit that selects, as an explanatory variable, a state index value having a high correlation with the change of the occupant to a specific state among a plurality of state index values indicating the identified states of the occupant.
A plurality of index value calculation units that calculate the state index value selected by the explanatory variable selection unit based on at least a part of the sensor information.
A score calculation unit that calculates a score indicating the possibility that the occupant is in a specific state for the state index value calculated by each of the plurality of index value calculation units.
A state determination device comprising: a score determination unit for determining that the occupant is in a specific state when the total score obtained by summing the scores is equal to or higher than a predetermined score threshold value. The score calculation unit is characterized in that it calculates the score based on a mathematical formula showing a correlation between the state index value of the identified occupant and the probability of being in the specific state. The state determination device described in 1. A receiver that receives collected data from the management device, and
Based on the collected data, a state threshold value, which is a threshold value for determining whether or not the occupant's state is the specific state, is calculated for each of the state index values selected by the explanatory variable selection unit. Threshold calculation unit and
A state determination unit for determining whether or not the state index value is equal to or higher than the state threshold value corresponding to the state index value is provided.
The collected data correlates the state index value of the occupant of the other moving body calculated in the other moving body with the state of the occupant of the other moving body determined in the other moving body. Contains multiple data
The state determination device according to claim 1 or 2, wherein the score calculation unit calculates a score for a state index value determined by the state determination unit to be equal to or higher than the state threshold value. The sensor information is data indicating the number of biological beats, and is
The state determination device according to any one of claims 1 to 3, wherein the index value calculation unit specifies the state index value by using the sensor information in a predetermined first period. Before the index value calculation unit calculates the state index value, it includes a sub-state determination unit that determines whether or not the occupant's state is a specific state based on at least a part of the sensor information. The state determination device according to claim 4, wherein the state determination device is characterized in that. It is provided with an average determination unit for determining whether or not the occupant is in a specific state based on the state index value or the value of the intermediate result when calculating the state index value from the sensor information.
In the average determination unit, when the state index value or the intermediate result value is equal to or higher than the average value of the occupant's state index values in the past second period, or the occupant's said in the past second period. The state determination device according to any one of claims 1 to 5, wherein the occupant is determined to be in a specific state when the average value of the intermediate results is equal to or higher than the average value. The state threshold calculation unit is
It is a value equal to or more than the median value of the state index values calculated in the other moving body, and
The state according to claim 3, wherein a value in which half or more of the data for which the occupant is determined to be in a specific state is used as the state index value in the moving body among the plurality of data. Judgment device. The invention according to any one of claims 1 to 7, further comprising a notification unit for notifying the occupant of a warning via an output device when the occupant is determined to be in a specific state. Status judgment device. The state determination device according to any one of claims 1 to 8, and the state determination device.
A state determination system comprising the sensor device. It is a control method of the state judgment device.
An acquisition step of acquiring sensor information about an occupant of the moving body from one or more sensor devices arranged in the internal space of the moving body.
An explanatory variable selection step for selecting as an explanatory variable a state index value having a high correlation with the change of the occupant to a specific state among a plurality of state index values indicating the identified states of the occupant.
An index value calculation step for calculating the state index value selected in the explanatory variable selection step based on at least a part of the sensor information, and an index value calculation step.
With respect to the state index value calculated in the index value calculation step, a score calculation step for calculating a score indicating the possibility that the occupant is in a specific state, and a score calculation step.
A control method comprising: a score determination step for determining that the occupant is in a specific state when the total score obtained by summing the scores is equal to or higher than a predetermined score threshold value.

Images