JP7405030B2 - Condition determination device, condition determination system, and control method - Google Patents

Description

The present disclosure relates to a state determination device, a state determination system, and a control method.

2. Description of the Related Art Conventionally, there is a technology for determining a human condition from biological information such as pulse rate. For example, Patent Document 1 describes a technique for determining whether a certain person is drowsy using a biological heart rate such as heartbeat or pulse as an index value. With this technology, if the difference between a person's pulse rate measured in advance when they are awake and the pulse rate measured in real time exceeds a threshold, it is determined that the person is drowsy. do.

Japanese Patent Application Publication No. 2015-195982

However, in the technology described in Patent Document 1, the conditions under which the biological heart rate was measured in advance, such as the human physical condition or environmental data such as the measurement time and temperature, are the same as the conditions when measuring the biological heart rate in real time. If the conditions were different, there was a risk that accurate determination could not be made. Further, since the biological heart rate measured in advance is used, there is a possibility that the determination accuracy may deteriorate even if a long period of time has passed since the measurement.

One aspect of the present disclosure is made in view of the above-mentioned problems, and aims to accurately determine a human condition.

In order to solve the above problem, a state determination device according to one aspect of the present invention continuously collects sensor information regarding an occupant of the moving body from one or more sensor devices arranged in the interior space of the moving body. an acquisition unit that acquires the information; an index value calculation unit that calculates a state index value indicating the condition of the occupant based on at least a portion of the sensor information; a reception unit that receives collected data from a management device; a state threshold calculating unit that calculates a state threshold that is a threshold for determining whether the state of the occupant indicated by the state index value is a specific state based on the state; a state determination unit that determines that the state of the occupant is in the specific state if the state is equal to or greater than a threshold; It is characterized in that it includes a plurality of data in which the state index value is associated with the state of the occupant of the other moving object determined in the other moving object.

According to the above configuration, the state determination device determines the current state according to the state index value of the other occupant calculated by the other moving body and the result of determining the state of the other occupant based on the state index value. A state threshold value is calculated as a reference for state determination. Therefore, the human condition can be accurately determined.

In the condition determining device, the sensor information may be data indicating a biological heart rate, and the index value calculation unit specifies the condition index value using the sensor information of a predetermined first period. It's okay.

According to the above configuration, the human condition can be determined more accurately based on the data of the biological heart rate during the predetermined first period.

The state determination device determines whether the state of the occupant is a specific state based on at least a part of the sensor information before the index value calculation unit calculates the state index value. It may also include a sub-state determining section.

According to the above configuration, until the index value calculation section obtains data on the biological beat rate for a period long enough to calculate the state index value (i.e., the predetermined first period), the sub-state determination section The condition of the occupant can be determined. Therefore, the human condition can be determined more quickly.

The state determining device performs an average determination for determining whether or not the occupant is in a specific state based on the state index value or a value of an intermediate result when calculating the state index value from the sensor information. If the condition index value or the value of the intermediate result is greater than or equal to the average value of the condition index values of the occupant in the past second period, or It may be determined that the occupant is in a specific state when the intermediate result value of the occupant in the second period is equal to or greater than the average value.

If the condition index value calculated by the index value calculation unit is greater than or equal to the average value of the occupant's own condition index values calculated in the past second period or the average value of intermediate results, the condition of the occupant is equal to or higher than the past average. The situation is likely to be different from the normal state. According to the above configuration, the state determining device can determine that the occupant is in a specific state when there is a high possibility that the occupant's state is different from normal.

In the state determination device, the state threshold calculation unit is configured to calculate a value that is equal to or greater than the median of the state index values calculated for the other moving body, and that the occupant is in a specific state among the plurality of data. A value for which half or more of the data is determined to be present may be used as the state index value for the mobile body. According to the above configuration, the human condition can be determined more accurately.

The state determination device includes an explanatory variable selection unit that selects, as an explanatory variable, a state index value that has a high correlation with a change to the specific state of the identified occupant; and a state index value selected by the explanatory variable selection unit. a score calculation unit that calculates a score indicating the possibility that the occupant is in a specific state with respect to the state index values calculated by each of the plurality of index value calculation units; The vehicle may further include a score determining unit that determines that the occupant is in a specific state when the total score obtained by summing the scores is equal to or higher than a predetermined score threshold.

According to the above configuration, a state index value having a high correlation with a change to a specific state of the occupant is selected, a score of the state index value is calculated, and a final state determination result is determined based on the score. can be lowered. Therefore, the human condition can be determined more accurately.

In the state determination device, the score calculation unit may calculate the score based on a formula that indicates a correlation between the state index value of the identified passenger and a probability of being in the specific state. . According to the above configuration, the human condition can be determined more accurately.

The condition determining device may include a notification unit that notifies the occupant of a warning via an output device when it is determined that the occupant is in a specific condition. According to the above configuration, it is possible to notify the occupant that the occupant is in a specific state.

In order to solve the above problem, a state determination system according to one aspect of the present invention includes the state determination device, the sensor device, and the management device. According to the above configuration, the same effects as those of the state determining device can be achieved.

In order to solve the above-mentioned problems, a control method for a state determination device according to one aspect of the present invention collects sensor information regarding an occupant of a moving body from one or more sensor devices arranged in an internal space of the moving body. an acquisition step of continuously acquiring; an index value calculation step of calculating a state index value indicating the condition of the occupant based on at least part of the sensor information; and a reception step of receiving collected data from a management device; a state threshold calculation step of calculating, based on the collected data, a state threshold that is a threshold for determining whether the state of the occupant indicated by the state index value is a specific state; and the state index value is equal to or greater than the state threshold, determining that the state of the occupant is in the specific state, and the collected data includes the state of the other moving body calculated in the other moving body. A plurality of pieces of data are included in which the state index value of the occupant is associated with the state of the occupant of the other moving body determined in the other moving body. According to the above configuration, the same effects as those of the state determining device can be achieved.

According to one aspect of the present disclosure, a human condition can be accurately determined.

1 is a block diagram showing a main part configuration of a state determination system according to Embodiment 1. FIG. FIG. 7 is a diagram schematically showing a method of calculating a second threshold value in a graph. FIG. 3 is a diagram showing a data structure of a status log. 7 is a flowchart showing the flow of processing in the state determination device according to the first embodiment. It is a flowchart which shows the flow of a process of a 1st determination. 7 is a flowchart showing the flow of second determination processing. FIG. 2 is a block diagram showing a main part configuration of a state determination system according to a second embodiment. 7 is a flowchart showing the flow of processing in the state determination device according to the second embodiment. It is a flowchart which shows the flow of a process of a 3rd determination. It is a graph showing the appearance frequency of a certain passenger's RRI difference in a second period. FIG. 3 is a block diagram showing the configuration of main parts of a state determination system according to a third embodiment. FIG. 3 is a diagram showing a data structure of explanatory variable data. It is a graph showing an overall threshold value, an individual threshold value, and a score formula of explanatory variables that have no correlation with changes in the occupant's condition. It is a graph showing an overall threshold value, an individual threshold value, and a score formula of explanatory variables that have a correlation with a change in a passenger's condition. 7 is a flowchart showing the flow of processing in the state determination device according to the third embodiment. FIG. 3 is a diagram schematically showing selection of explanatory variables used to determine the state of a certain occupant and results of state determination using the explanatory variables. FIG. 7 is a block diagram showing the main part configuration of a state determination system according to a fourth embodiment.

[Embodiment 1]
≪System overview≫
The state determination system 100 according to the present embodiment is a system for determining the state of an occupant staying in the interior space of a moving body. Here, "state" refers to a human's physiological or emotional state. For example, the state determination system 100 can determine whether the passenger is sleepy, angry, excited, etc. as the "state" of the passenger. In this embodiment, as an example, a state determination system 100 for determining whether or not an occupant is drowsy will be described with reference to FIGS. 1 to 6.

≪Main part configuration≫
FIG. 1 is a block diagram showing the main configuration of a state determination system 100 according to this embodiment. The state determination system 100 includes a state determination device 1A, a sensor group 2, a storage device 3, a server (management device) 4, and an output device 5. Note that the storage device 3 is not an essential component.

In the example of FIG. 1, the state determination device 1A, the sensor group 2, the storage device 3, and the output device 5 are mounted on a moving body 6. Although FIG. 1 shows only the internal structure of one moving body 6 in detail for the sake of simplicity, all the moving bodies 6 shown in FIG. 1 have the same configuration. Each sensor included in the sensor group 2 and the state determination device 1A are communicably connected. The storage device 3 and the state determination device 1A are communicably connected. The state determination device 1A and the server 4 are communicably connected. Note that, as shown in FIG. 1, the server 4 communicates with a plurality of mobile bodies 6.

(Mobile object 6)
The moving body 6 is a moving body having a space inside, such as a car, a train, or an airplane. Hereinafter, when it is simply referred to as "internal space", it refers to the space inside the moving body 6. In addition, people staying in the interior space are simply referred to as "occupants."

(Sensor group 2)
Sensor group 2 is a group of sensors for measuring sensor information. Here, "sensor information" refers to all information regarding the occupant that can be obtained by the sensor group 2. For example, the sensor information may be data regarding the occupant's appearance, such as the movement of the occupant's whole body or a part of the body, and facial expressions. Further, the sensor information may be information indicating vital data of the occupant, such as pulse, respiration, body temperature, blood pressure, consciousness, and reflexes. In this embodiment, the sensor information refers to data of an image taken by the camera (sensor device) 21 and measurement data of the pulse sensor (sensor device) 22.

The sensor group 2 consists of one or more sensors arranged in the space inside the moving body 6. The various sensors are connected to the state determining device 1A by wire or wirelessly, and transmit the sensor information acquired by each sensor to the state determining device 1A. The types of sensors included in sensor group 2 are not particularly limited. For example, the sensor group 2 includes a camera 21 and a pulse sensor 22 as output devices.

The camera 21 photographs at least a portion of the occupant's body. Note that the body referred to here also includes the face of the passenger. The captured image of the camera 21 is transmitted to the state determination device 1A as sensor information. Note that the image taken by the camera 21 may be a still image or a moving image. Hereinafter, "image" and "captured image" refer to both still images and moving images. When the camera 21 takes still images, the time interval between the images is not particularly limited, but it is preferable that the images are taken at a time interval that allows the movements of the occupant to be tracked in real time. The camera 21 starts photographing at a predetermined timing, and thereafter continuously photographs the occupant. The predetermined timing may be, for example, the timing when a passenger takes a seat in the moving object 6.

Further, the specific configuration of the camera 21 and its position within the predetermined space are not particularly limited. For example, when the moving object 6 is a passenger car, the camera 21 may be placed in front of a seat of the car at a position and angle such that the face of the person sitting in the seat is reflected. Thereby, the camera 21 can acquire an image capturing the face of the driver or passenger of the passenger car from the front. In other words, the camera 21 can capture an image in which the eye movements, eyelid movements, facial expressions, etc. of the driver or passenger can be identified.

The pulse sensor 22 is a sensor that measures human pulse rate. The pulse sensor 22 may be, for example, an electrode-type pulse sensor disposed on the armrest of the seat of the moving body 6. Further, the pulse sensor 22 may be a radio wave pulse sensor built into the seat of the moving body 6. More specifically, for example, the pulse sensor 22 may be a vital sensor that detects a heartbeat signal from a millimeter wave radar signal. Measurement data from the pulse sensor 22 is transmitted as sensor information to the state determining device 1A. The pulse sensor 22 starts measuring the pulse at a predetermined timing, and then continuously measures the passenger's pulse. The predetermined timing may be, for example, the timing when a passenger takes a seat in the moving object 6.

Note that the sensor group 2 may include sensors that measure biological heart rate other than pulse rate, such as heartbeat. In addition, the sensor group 2 may include sensors that measure vital data such as respiration, body temperature, blood pressure, consciousness, and reflexes. The measurement data of the sensor is also transmitted to the state determination device 1A as sensor information.

Note that a plurality of sensors of the same type may be arranged at different positions in the interior space of the moving body 6. For example, when the moving object 6 is a passenger car, one or more cameras 21 may be arranged in front of each seat of the passenger car. With this, for example, when a person sits on each seat, images of each person can be acquired. Further, one or more pulse sensors 22 may be arranged in each seat of the passenger car. This makes it possible to measure the pulse of each seat, that is, the pulse of each person.

(Status determination device 1A)
The condition determining device 1A determines the condition of the occupant based on the sensor information acquired from the sensor group 2 and the collected data acquired from the server 4. Details of the collected data will be described later. The state determination device 1A includes a data acquisition unit (acquisition unit) 11, a first determination unit (sub-state determination unit) 12, a communication unit (reception unit) 13, a second determination unit 14, a notification unit 15, has.

The state determination device 1A according to this embodiment mainly performs state determination using measurement data from the pulse sensor 22. However, in order to accurately identify the condition of the passenger based on changes in the passenger's pulse, it is desirable to measure the pulse for a certain period of time (for example, 5 minutes) using the pulse sensor 22.

Therefore, in the state determination device 1A according to the present embodiment, the state can be determined in a short measurement time compared to the pulse sensor 22 during the period from the start of measurement by the pulse sensor 22 until the measurement data of the pulse sensor 22 is sufficiently accumulated. Determine the status using another determination method if possible. Specifically, the state determining device 1A determines whether or not the occupant is drowsy using images captured by the camera 21.

Hereinafter, in the state determination device 1A, the first state determination performed after the start of measurement (or imaging) of the sensor group 2 will be referred to as a first determination. Further, hereinafter, in the state determination device 1A, the state determination performed after the first determination will be referred to as a second determination. Note that the first determination is merely an additional configuration for performing state determination earlier. Therefore, the configuration related to the first determination is not an essential configuration in this embodiment.

The data acquisition unit 11 acquires sensor information of the mobile object 6 from the sensor group 2 . The data acquisition unit 11 outputs at least part of the acquired sensor information to the first determination unit 12. Further, the data acquisition unit 11 outputs at least a part of the acquired sensor information to the second determination unit 14. In this embodiment, the data acquisition unit 11 outputs the captured image of the camera 21 to the first determination unit 12 and outputs the pulse measurement data of the pulse sensor 22 to the second determination unit 14 . Note that the data acquisition unit 11 may continuously acquire sensor information from the sensor group 2 at predetermined time intervals or in real time.

The communication unit 13 performs communication between the state determination device 1A and the server 4. For example, the communication unit 13 receives collected data from the server 4 and outputs it to the second determination unit 14. For example, the communication unit 13 transmits the status log 32 input from the second determination unit 14 to the server 4.

The first determination unit 12 performs a first determination based on sensor information. The first determination unit 12 first calculates a first index value based on an image taken by the camera 21.

In this embodiment, the first determination unit 12 calculates the drowsiness level based on the drowsiness evaluation method of NEDO (New Energy and Industrial Technology Development Organization) as the first index value. According to the NEDO sleepiness evaluation method, the sleepiness level is calculated as a level from 1 to 5. Sleepiness level 1 is not sleepy at all, sleepiness level 2 is slightly sleepy, sleepiness level 3 is sleepy, sleepiness level 4 is quite sleepy, and sleepiness level 5 is very sleepy or just starting to fall asleep. Indicates the state in which

Next, the first determination unit 12 determines whether the calculated first index value is greater than or equal to the first threshold value. When the first index value is greater than or equal to the first threshold, the first determination unit 12 determines that the occupant is in a specific state. On the other hand, if the first index value is less than the first threshold, the first determination unit 12 determines that the occupant's condition is not in the specific condition.

In this embodiment, the first determination unit 12 sets the first threshold value of the sleepiness level to 3, as an example. If the calculated drowsiness level is 3 or more, it is determined that the occupant is drowsy, and if it is less than 3, it is determined that the occupant is not drowsy. The threshold value of the sleepiness level may be determined as appropriate depending on the level of sleepiness that is acceptable as "no sleepiness".

The first determination unit 12 notifies the notification unit 15 of the result of the state determination. In the case of the present embodiment, the first determination unit 12 notifies the notification unit 15 whether it is determined that the occupant is drowsy or not.

The second determination unit 14 uses the measurement data of the pulse sensor 22 to perform a second determination. The second determination unit 14 includes a second index value calculation unit (index value calculation unit) 141, a threshold calculation unit (state threshold calculation unit) 142, and a comparison determination unit (state determination unit) 143.

The second index value calculation unit 141 calculates a second index value (state index value) for determining the state based on the measurement data of the pulse sensor 22. The second index value calculation unit 141 outputs the calculated second index value to the comparison determination unit 143.

If the index value can be calculated from information from any of the sensor group 2 and can be used to determine the condition of the occupant, the type, calculation method, and value of the second index value can be determined. The unit is not particularly limited. For example, the second index value calculation unit 141 may calculate RRI (pulse interval time, RR Interval) from the measurement data of the pulse sensor 22, and calculate the second index value based on the RRI.

For example, the second index value calculation unit 141 may calculate the RRI difference as the second index value. "RRI difference" generally refers to the value of the difference between a certain RRI value and an RRI value immediately before the certain RRI in time series. Since RRI increases when tension eases, it can be said that the RRI difference also increases. Therefore, it can be said that the RRI difference also increases when sleepiness increases.

In this embodiment, the second index value calculation unit 141 calculates the second index value using the following method. The second index value calculation unit 141 calculates the RRI from the measurement data of the pulse sensor 22. Calculation of RRI is performed continuously while acquisition of measurement data continues. In parallel with the calculation of the RRI, the second index value calculation unit 141 divides the measurement data into units of time, and calculates the average value of the RRI for each division as needed. Hereinafter, the average value of RRI per unit time will be referred to as "average RRI." The unit time referred to here may be, for example, about 3 minutes.

Then, the second index value calculation unit 141 calculates the RRI difference by finding the difference between the latest average RRI and the previous average RRI. Therefore, the "RRI difference" in this embodiment indicates the difference between the latest average RRI calculated by the second index value calculation unit 141 and the average RRI immediately before the average RRI.

Note that the second index value calculation unit 141 continuously calculates the RRI difference at predetermined time intervals while the sensor information from the data acquisition unit 11 continues to be supplied. Then, the second index value calculation unit 141 outputs the calculated RRI difference to the successive comparison determination unit 143. The time interval at which the second index value calculation unit 141 calculates the RRI difference is not particularly limited, but it is desirable that the time interval be such that the second determination unit 14 can identify the presence or absence of drowsiness of the occupant without delay.

The threshold value calculation unit 142 calculates a second threshold value (state threshold value) that serves as a criterion for the second determination from the state log of collected data received from the server 4. The threshold calculation unit 142 outputs the calculated second threshold to the comparison determination unit 143. Note that the time interval at which the second index value calculation unit 141 calculates the second index value and the time interval at which the threshold value calculation unit 142 calculates the second threshold may be the same or different. .

More specifically, the threshold calculation unit 142 first refers to the user information 31 and extracts the status log used for calculating the second threshold from among the status logs included in the collected data received by the second determination unit 14. . Note that if the storage device 3 does not store the user information 31, if the status log does not include occupant attribute information, or if the status determination device 1A does not identify the individual occupant, the threshold calculation unit 142 No need to perform extraction. That is, the threshold calculation unit 142 may use all the status logs included in the collected data to calculate the second threshold.

The threshold calculation unit 142 then totals the second index values of each of the extracted status logs (that is, the second index values of other people), and calculates the median of the second index values of the other people. Furthermore, the threshold calculation unit 142 refers to the determination results of each of the extracted status logs, and calculates a second index value at which the proportion of logs determined to be drowsy among the extracted status logs is more than half (50%). Identify the range. Finally, the threshold calculation unit 142 determines, as the second threshold, the minimum second index value that is equal to or greater than the calculated median value and for which the proportion of state logs determined to be drowsy is 50% or more. . Note that the threshold value calculation unit 142 calculates the mode value instead of the median value of the second index values of other people, and the percentage of state logs in which the second index value is equal to or higher than the mode value and is determined to be drowsy becomes 50% or more. The minimum second index value may be determined as the second threshold value.

(Threshold calculation method)
FIG. 2 is a graph schematically showing a method of calculating the second threshold value in the threshold value calculation unit 142. Note that the graph in FIG. 2 is for visually explaining the method of calculating the second threshold in the threshold calculation unit 142, and there is no need for the threshold calculation unit 142 to actually create the illustrated graph.

The x-axis of the graph shows the second index value, ie, the RRI difference. The vertical axis of the graph indicates the number of status logs indicating each RRI difference (that is, the distribution of the second index values included in the collected data). In addition, in the example in Figure 2, the bar graph displayed in white indicates the distribution of RRI differences of the extracted condition logs, and the bar graph displayed in gray indicates the ``drowsiness'' in the second judgment of the extracted condition logs. 11 shows the distribution of RRI differences of status logs that are determined to be “Yes”.

In the illustrated example, the median value of the RRI differences indicated by the extracted status logs is 0.100. The proportion of state logs whose determination result is "drowsy" becomes 50% or more of all extracted state logs when the RRI difference is 0.125 or more. In other words, the total area of the gray bar graph exceeds 50% of the total area of the white bar graph when the RRI difference is 0.125 or more. Therefore, the threshold calculation unit 142 determines the second threshold to be 0.125.

The comparison/determination unit 143 determines the condition of the occupant based on the magnitude relationship between the second index value and the second threshold value. The comparison determination unit 143 determines whether the input second index value is greater than or equal to the input second threshold value. When the second index value is greater than or equal to the second threshold, the comparison determination unit 143 determines that the occupant is in a specific state. On the other hand, when the second index value is less than or equal to the second threshold value, the comparison determination unit 143 determines that the condition of the occupant is normal. The comparison and determination section 143 outputs the determination result to the notification section 15.

The notification unit 15 causes the output device 5 to output a first notification when the first determination unit 12 determines that the occupant is in a specific state. Further, the notification unit 15 causes the output device 5 to output a second notification when the second determination unit 14 (more specifically, the comparison determination unit 143) determines that the occupant is in a specific state. The contents of the first notification and the second notification are not particularly limited. For example, the first notification and the second notification may be warnings to the occupant.

(Output device 5)
The output device 5 outputs various notifications in the form of audio or video based on control instructions from the notification unit 15. The output device 5 is realized by, for example, a speaker or a display. Note that there may be a plurality of output devices 5. For example, the moving body 6 may include two or more speakers as the output device 5. For example, the moving body 6 may include both a speaker and a display as the output device 5.

(Storage device 3)
The storage device 3 stores data necessary for the operation of the state determining device 1A. For example, the storage device 3 stores user information 31 and a status log 32. Note that in this embodiment, the user information 31 is not essential information.

The user information 31 indicates personal information of the occupant of the mobile object 6. The user information 31 may be, for example, information such as the passenger's gender, age, and area of residence.

FIG. 3 is a diagram showing the data structure of the status log 32. The status log 32 is log data that shows the history of the second index value calculated by the second index value calculation unit 141 and the determination result of the second determination of the second index value. Further, the status log 32 may include identification information for uniquely identifying the occupant.

In the example of FIG. 3, the status log 32 has a "time" column, an "identification information" column, an "RRI difference" column, and a "judgment result" column. One record of the status log 32 in FIG. 3 indicates a second index value and a second determination result for a certain occupant at a certain time.

The "time" column stores the time when the second determination unit 14 generated each record. Note that the time may be measured using a timer (not shown) built into the state determining device 1A. Further, the "time" column may store the time when the second index value was calculated or the time when the second determination was made.

The "identification information" column stores passenger identification information (for example, passenger ID). Note that the method for identifying individual occupants is not particularly limited. For example, the state determining device 1A may identify the occupant by analyzing an image taken by the camera 21. Alternatively, the state determination device 1A may allow the occupant to input identification information of the occupant via an input device (not shown), and identify the individual occupant from the identification information.

The “RRI difference” column stores the RRI difference (that is, the second index value) calculated by the second index value calculation unit 141. The “determination result” column stores the result of the second determination made by the comparison determination unit 143. In the example of FIG. 3, the record corresponding to the RRI difference where the comparison/determination unit 143 has determined that "occupant is not drowsy" is "no drowsiness", and the record corresponding to the RRI difference where it is determined that "occupant is drowsy" is Information “feeling sleepy” is stored.

Note that, as described above, the second threshold value serving as the reference for the second determination is calculated each time by the threshold value calculation unit 142. Therefore, even if the RRI difference is the same value, the result of the second determination may be different, as in the record on the third line and the record on the fourth line in FIG. 3.

(Server 4)
The server 4 collects and manages status logs from the multiple status determination devices 1A. The server 4 includes a server communication section 41, a server control section 42, and a server storage section 43.

The server communication unit 41 performs communication between the server 4 and the state determination device 1A. The server storage unit 43 stores various data necessary for the operation of the server 4. The server storage unit 43 stores collected data 431. The collected data 431 is data that summarizes state logs collected from the state determination devices 1A of a plurality of moving bodies 6.

Note that the server control unit 42 may delete records for which a predetermined period of time has elapsed from among the status log records included in the collected data 431. For example, records for which one hour has elapsed since the time indicated by each record may be deleted.

Thereby, the capacity of the collected data 431 can be reduced. Further, as a result, the collected data 431 always includes only the status log for the most recent predetermined period. The specific length of the "most recent predetermined period" is not particularly limited, but may be, for example, about one hour.

Thereby, when calculating the second threshold value, it is possible to always calculate the second threshold value based on the status logs collected within a predetermined period going back from the current time. Therefore, by appropriately setting the predetermined period, even if environmental conditions that change over time, such as weather and temperature, change, status logs generated by other mobile units 6 under approximately the same conditions can be read. can be used to calculate the second threshold value. Therefore, it is possible to reduce the possibility that an erroneous determination due to a difference in environmental conditions will occur in the second determination. Therefore, the accuracy of the second determination is improved.

The server control unit 42 manages the server 4 in an integrated manner. For example, the server control unit 42 receives a status log record from the status determination device 1A via the server communication unit 41. The server storage unit 43 adds the received state log for one record to the collected data 431. As shown in FIG. 1, the server 4 similarly collects status log records in real time from the status determination devices 1A of a plurality of moving bodies 6. Thereby, in the collected data 431 of the server 4, the calculation results of the RRI difference and the determination results of the presence or absence of drowsiness in each state determination device 1A are collected and accumulated in real time.

≪Processing flow≫
FIG. 4 is a flowchart showing the flow of processing in the state determining device 1A. In FIG. 4, the time when measurement of various information in the sensor group 2 is started is shown as the start of the process.

When the various sensors of the sensor group 2 start measuring and acquiring various information, the data acquisition unit 11 acquires various sensor information from the sensor group 2 (S11). For example, the data acquisition unit 11 acquires a captured image from the camera 21 and pulse measurement data from the pulse sensor 22. Of these acquired data, the data acquisition unit 11 outputs the image captured by the camera 21 to the first determination unit 12 and outputs the measurement data of the pulse sensor 22 to the second determination unit 14 . When sensor information used for the second determination (in this embodiment, measurement data of the pulse sensor 22) is input, the second determination section 14 generates sufficient sensor information for the second determination section 14 to perform the second determination. is acquired (S12).

Here, "sensor information sufficient to perform the second determination" is sensor information for a predetermined period (first period) that allows the second index value calculation unit 141 to calculate the second index value. be. In this embodiment, if the second index value calculation unit 141 can acquire sensor information for a period enough to calculate the average RRI at least twice, it is possible to calculate the RRI difference. Therefore, the sensor information sufficient for making the second determination is at least the measurement data of the pulse sensor 22 for a period twice as long as the average RRI calculation time interval.

If sufficient sensor information has not been acquired to make the second determination (NO in S12), the second determination unit 14 instructs the first determination unit 12 to perform the first determination. Upon receiving the instruction, the first determination unit 12 executes a first determination based on the captured image input from the data acquisition unit 11 (S13). On the other hand, if sensor information sufficient for the second determination has been acquired (YES in S12), the second determination unit 14 executes the second determination (S16).

For example, when the data acquisition unit 11 acquires sensor information for the first time after the state determination device 1 is started, the second determination unit 14 has acquired only the first sensor information, so the determination in S12 is NO, and the first determination is A first determination by the unit 12 is performed.

(First judgment)
FIG. 5 is a flowchart showing the flow of the first determination process. The first determination unit 12 first calculates the drowsiness level based on the image taken by the camera 21 (S131). Next, the first determination unit 12 determines whether the calculated sleepiness level is 3 or higher (S132). If the drowsiness level is 3 or higher (YES in S132), the first determination unit 12 determines that the occupant is drowsy (S133). On the other hand, if the sleepiness level is less than 3 (NO in S132), the first determination unit 12 determines that the occupant is not sleepy (S134).

This will be explained with reference to FIG. 4 again. When the first determination is completed (S13), the first determination section 12 notifies the notification section 15 of the determination result.

The notification unit 15 determines whether or not to cause the output device 5 to output the first notification based on the determination result of the first determination unit 12 (S14). In the first determination, if it is determined that the occupant is drowsy (YES in S14), the notification unit 15 causes the output device 5 to output the first notification (S15). On the other hand, if it is determined that the occupant is not sleepy (NO in S14), the notification unit 15 ends the process without causing the output device 5 to output the first notification. Next, when the data acquisition unit 11 acquires various sensor information, the state determination device 1A executes the process again from S11. If the second determination unit 14 is able to acquire enough sensor information for the second determination (YES in S12), the second determination is executed (S16).

(Second judgment)
FIG. 6 is a flowchart showing the flow of the second determination process. The second index value calculation unit 141 of the second determination unit 14 calculates the second index value, that is, the RRI difference, based on the measurement data of the pulse sensor 22 (S161). The second index value calculation unit 141 outputs the RRI difference to the comparison determination unit 143.

On the other hand, the second determination unit 14 receives collected data from the server 4 via the communication unit 13 (S162). The threshold calculation unit 142 of the second determination unit 14 refers to the user information 31 and extracts a status log to be used for calculating the second threshold from among the status logs included in the collected data received by the second determination unit 14. (S163). Note that the process of S163 is not essential. When not extracting state logs, the threshold calculation unit 142 may use all state logs included in the collected data in subsequent processing.

The threshold calculation unit 142 then totals the second index values of each of the extracted status logs (that is, the second index values of other people), and calculates the median of the second index values of the other people (S164). Furthermore, the threshold calculation unit 142 refers to the determination results of each of the extracted status logs, and specifies a value of the second index value for which the ratio of logs determined to be drowsy among the extracted status logs is 50% or more. (S165). Finally, the threshold calculation unit 142 determines, as the second threshold, the minimum second index value that is equal to or greater than the calculated median value and for which the proportion of logs determined to be drowsy is 50% or more. S166). The threshold value calculation unit 142 outputs the determined second threshold value to the comparison determination unit 143.

The comparison determination unit 143 determines whether the RRI difference calculated by the second index value calculation unit 141 is greater than or equal to the second threshold (S167). If the RRI difference is greater than or equal to the second threshold (YES in S167), the comparison determination unit 143 determines that the occupant is sleepy (S168). On the other hand, if the RRI difference is less than the second threshold (NO in S167), the comparison determination unit 143 determines that the occupant is not sleepy (S169).

This will be explained with reference to FIG. 4 again. When the second determination is completed (S16), the comparison determination unit 143 notifies the notification unit 15 of the determination result. The notification unit 15 determines whether or not to cause the output device 5 to output the second notification based on the determination result of the comparison determination unit 143 (S17). In the second determination, if it is determined that the occupant is drowsy (YES in S17), the notification unit 15 causes the output device 5 to output a second notification (S18). On the other hand, if it is determined that the occupant is not sleepy (NO in S17), the notification unit 15 ends the process without causing the output device 5 to output the second notification.

Next, when the data acquisition unit 11 acquires various sensor information, the state determination device 1A executes the process again from S11. Since the measurement data of the pulse sensor 22 is sequentially supplied to the data acquisition unit 11, the second determination unit 14 is in a state in which sensor information sufficient for the second determination is accumulated. Therefore, the second determination unit 14 repeatedly performs the second determination (S16) every time it acquires sensor information from the data acquisition unit 11.

Note that, after making the second determination (S16), the second determination unit 14 may create a status log and store it in the storage device 3 as one record of the status log 32. Further, the second determination unit 14 may transmit the created status log (that is, the status log for one record) to the server 4 via the communication unit 13. The server storage unit 43 of the server 4 receives one record worth of status logs via the server communication unit 41. The server storage unit 43 adds the received state log for one record to the collected data 431.

Note that if the first determination is not performed, the state determination device 1A does not perform the processes of S13 to S15 in FIG. 4, and continues until sufficient measurement data is obtained to perform the second determination (YES in S12). You can wait. Alternatively, the second determination may be performed even if sensor information for a period sufficient to perform the second determination has not been obtained (S16). For example, if sensor information for a sufficient period of time has not been obtained to perform the second determination, the second determination unit 14 may assume that the second index value is 0 and perform the second determination. .

(Modified example)
In this embodiment, the types of the first index value and the second index value and the method of calculating the values may be determined as appropriate depending on the data that can be obtained from the sensor group 2. That is, the first determination unit 12 and the second determination unit 14 may each calculate an index value that can be calculated using information obtained from the sensor group 2 as the first index value or the second index value. Then, the data acquisition unit 11 may appropriately supply the first determination unit 12 and the second determination unit 14 with information required by the first determination unit 12 and the second determination unit 14.

For example, the threshold calculation unit 142 of the second determination unit 14 may calculate the following value as the second index value. That is, the threshold calculation unit 142 may calculate the pulse rate per unit time as the second index value. Further, the threshold calculation unit 142 may calculate at least one of the maximum change range of RRI per unit time, the average value of RRI per unit time, and the median value of RRI per unit time as the second index value. . Further, the threshold calculation unit 142 may calculate one of the integrated values of the high frequency component, low frequency component, very low frequency component, and very low frequency component of RRI per unit time as the second index value. Further, the threshold calculation unit 142 may calculate the ratio of the integral values of different frequency components of the RRI, such as the ratio of the integral values of the high frequency component and the low frequency component of the RRI, as the second index value. Further, the threshold calculation unit 142 may calculate a coefficient of simple regression of RRI as the second index value.

Further, for example, the second determination unit 14 may analyze the captured image of the camera 21 and calculate the second index value based on the analysis result. For example, the threshold calculation unit 142 may identify the movement of the passenger's eyelids from the captured image. Specifically, the threshold calculation unit 142 may calculate the eye-closing time, the number of eye-closing or blinking per unit time, etc. as the second index value.

[Embodiment 2]
The condition determination system 100 may consider the past second index value of the occupant when determining the condition of the occupant. A second embodiment of the present disclosure will be described below. For convenience of explanation, members having the same functions as those described in Embodiment 1 will be denoted by the same reference numerals, and the description will not be repeated.

≪Main part configuration≫
FIG. 7 is a block diagram showing the main configuration of the state determination system 200 according to this embodiment. Condition determination system 200 differs from condition determination system 100 in that it includes condition determination device 1B. The state determining device 1B differs from the state determining device 1A in that it includes a third determining section (average determining section) 16.

After calculating the second index value, the second index value calculation unit 141 of the second determination unit 14 according to the present embodiment outputs the second index value to the third determination unit 16. The notification unit 15 according to the present embodiment causes the output device 5 to output a third notification when the third determination unit 16 determines that the occupant is in a specific state. The content of the third notification may be a warning to the occupant, similar to the first notification and the second notification.

The third determination unit 16 performs a third determination based on the second index value. The third determination is to compare the average value of the past second index values of the occupants of the moving object 6 with the latest second index value obtained from the second index value calculation unit 141. This is a determination method for determining whether or not is in a specific state. Details of the third determination will be described later. The third determination unit 16 notifies the notification unit 15 of the determination result.

≪Processing flow≫
FIG. 8 is a flowchart showing the flow of processing in the state determining device 1B according to this embodiment. Note that in FIG. 8, the steps with the same step numbers as in FIG. 4 are the same processes as in FIG. 4, and therefore the description will not be repeated.

The state determination device 1B executes a third determination in parallel with the second determination (S19). Note that, as described above, the second index value is used for the third determination. Therefore, the third determination unit 16 acquires the second index value from the second index value calculation unit 141 at least once, and then executes the third determination. Therefore, when the second determination is performed for the first time, the third determination may not be performed in parallel.

FIG. 9 is a flowchart showing the flow of the third determination process. When the third determination unit 16 acquires the second index value, that is, the RRI difference (S191), it reads out the occupant status log for a predetermined second period. The "second period" referred to here is, for example, about 10 hours going back from the current time. Next, the third determination unit 16 calculates the average value of the RRI differences indicated by each record of the read status log (S192). Subsequently, the third determination unit 16 calculates the difference between the RRI difference calculated by the second index value calculation unit 141 (that is, the current RRI difference of the occupant) and the average value of the RRI differences of the condition logs for the second period +3σ. The size is compared (S193). Note that "3σ" is a value that determines the allowable range of error from the average value.

If the RRI difference calculated by the second index value calculation unit 141 is equal to or greater than the average value of the RRI differences of the state logs for the second period + 3σ (YES in S193), the third determination unit 16 determines that the occupant is sleepy, It is determined that (S194). If the RRI difference calculated by the second index value calculation unit 141 is less than the average value of the RRI differences of the state logs for the second period + 3σ (NO in S193), the third determination unit 16 determines that the occupant is not drowsy. It is determined that (S195).

Note that addition of the error 3σ to the average value is not essential. That is, in S193, the third determination unit 16 may compare and determine the RRI difference and the average value, instead of the RRI difference calculated by the second index value calculation unit 141 and the average value +3σ. If the RRI difference is greater than or equal to the average value, it may be determined that the occupant is drowsy, and if it is less than the average value, it may be determined that the occupant is not drowsy.

The explanation will be given by returning to FIG. 8 again. When the third determination is completed (S19), the third determination unit 16 notifies the notification unit 15 of the determination result. The notification unit 15 determines whether or not to cause the output device 5 to output the third notification based on the determination result of the third determination unit 16 (S20). In the third determination, if it is determined that the occupant is drowsy (YES in S20), the notification unit 15 causes the output device 5 to output a third notification (S21). On the other hand, if it is determined that the occupant is not sleepy (NO in S20), the notification unit 15 ends the process without causing the output device 5 to output the third notification.

If the currently calculated second index value is larger than the average value of the occupant's own second index value, that is, the value estimated to be the occupant's second index value during normal times, it is determined that the occupant's condition is normal. It is likely that they are different. According to the above configuration, the state determining device can determine that the occupant is in a specific state when there is a high possibility that the occupant's state is different from normal.

≪Frequency of appearance of RRI difference and third judgment≫
FIG. 10 is a graph showing the appearance frequency of a certain passenger's RRI difference in the second period. Note that the graph in FIG. 10 is for visually explaining the determination process in the third determination unit 16, and there is no need for the third determination unit 16 to actually create the illustrated graph.

The frequency of appearance of individual RRI differences usually converges to a shape similar to a normal distribution as shown in FIG. As shown in FIG. 10, the farther a value is from the average value in the graph, the lower the frequency of appearance. Therefore, it can be said that the farther the current RRI difference of the occupant is from the average value shown in FIG. 10, the higher the possibility that the occupant is in a different state from normal.

Therefore, by adjusting the value of the error 3σ, it is possible to adjust the threshold value of the appearance frequency for determining whether the RRI difference of a certain occupant is determined to be “normal” or “abnormal”. For example, in FIG. 10, it is assumed that the value of the average RRI difference +3σ is set to a value where the appearance frequency shown on the vertical axis is 5%. Under this assumption, if the current RRI difference of the occupant calculated by the second index value calculation unit 141 is greater than or equal to the average value of RRI differences + 3σ, the appearance frequency of the current RRI difference is 5% or less (the graph in FIG. 10 (values within the shaded area). Therefore, it can be determined that the current RRI difference is an abnormal value that is normally difficult to calculate for the occupant.

In this way, the third determination unit 16 determines whether or not the occupant is drowsy by performing the third determination to determine whether or not the RRI difference has shown an abnormal value for the individual occupant. According to this third determination, it is possible to adjust how far the RRI difference deviates from the average value to be determined as abnormal, depending on the magnitude of the "3σ" value added to the average value.

The third determination unit 16 according to the present embodiment may count the number of times the second index value calculated by the second index value calculation unit 141 shows an abnormal value. Then, if the number of times the second index value shows an abnormal value is less than a predetermined number of times, the third determination unit 16 determines that the occupant is not in a specific state, and the number of times the second index value shows an abnormal value is a predetermined number of times. In the above case, it may be determined that the occupant is in a specific state. Note that this count may be reset to 0 when the third determination unit 16 makes a determination once.

The specific number of "predetermined times" is not limited. For example, the predetermined number of times may be 10 times. In this way, when an abnormal value is detected a certain number of times or more, it is determined that the occupant is in a specific state, and when the value temporarily increases significantly, the third determination unit 16 detects an incorrect value. It is possible to prevent a judgment from being made. Therefore, the accuracy of the third determination can be improved.

Note that, when counting the above-mentioned "number of times an abnormal value is shown", the third determination unit 16 may set a time limit until the count is reset. That is, the third determination unit 16 may determine that the condition of the occupant is abnormal if the second index value is equal to or greater than the average value +3σ a predetermined number of times or more within a predetermined time. Then, when the predetermined time has elapsed, a count of the number of times the second index value was the average value +3σ may be reset. Thereby, the accuracy of the third determination can be further improved.

The third determination unit 16 according to the present embodiment may use, for the third determination, a value that is an intermediate result when calculating the second index value from the sensor information, instead of the second index value of the occupant. Note that when using the value of the intermediate result for the third determination, the second determination unit 14 uses the status log 32 that includes the history of the value of the intermediate result instead of the second index value or together with the second index value. Create.

Specifically, for example, the third determination unit 16 may perform the third determination using the average RRI instead of the RRI difference. When performing the third determination using the average RRI, the second determination unit 14 creates a status log 32 including the history of the average RRI, and stores it in the storage device 3. Further, when performing the third determination using the average RRI, the second index value calculation unit 141 outputs the average RRI to the third determination unit 16. The subsequent processing may be executed by replacing the "RRI difference" in the processing described in this embodiment with the "average RRI."

[Embodiment 3]
The state determination system 300 according to the present embodiment, like the state determination systems 100 and 200, is a system for determining the state of an occupant staying in the interior space of a moving body. There are various methods for determining whether a person is in a specific state such as drowsiness using an index value indicating the state. However, an effective determination method, that is, an index value that is highly correlated with the above-mentioned "specific condition" differs from person to person.

The condition determination system 300 according to this embodiment performs condition determination in consideration of the above-mentioned individual differences. Specifically, the condition determination system 300 calculates a plurality of types of index values for specifying the condition of the occupant using a plurality of methods. Next, the condition determination system 300 selects an index value with a high correlation for each occupant, and calculates a score indicating the possibility that the occupant is in a specific condition according to the value of the index value. Then, the state determination system 300 determines whether the sum of the calculated scores is greater than or equal to a predetermined third threshold (score threshold). If the total score is equal to or greater than the third threshold, the state determination system 300 makes a final determination that the occupant is in a specific state.

The third embodiment of the present disclosure will be described in detail below using FIGS. 11 to 16. Note that members having the same functions as those described in Embodiment 1 or 2 will be denoted by the same reference numerals, and the description will not be repeated.

≪Main part configuration≫
FIG. 11 is a block diagram showing the main configuration of the state determination system 300 according to this embodiment. The state determination system 300 does not need to include the server 4, unlike the state determination systems 100 and 200.

The state determination system 300 includes a state determination device 1C. Further, the storage device 3 related to the state determination system 300 stores user information 31, a state log 32, and explanatory variable data 33. Note that in this embodiment, the user information 31 and the status log 32 are not essential data.

(Status determination device 1C)
The state determination device 1C does not need to include the communication unit 13. Moreover, the state determination device 1C has a plurality of explanatory variable calculation units 17 and a plurality of score calculation units 18 corresponding to the explanatory variable calculation units 17 instead of the second determination unit 14. Different from 1A and 1B. Although not included in the example of FIG. 11, the state determining device 1C may include the first determining section 12. In the example of FIG. 11, the state determination device 1C includes three explanatory variable calculation units 17A to 17C as examples of the plurality of explanatory variable calculation units 17. Further, in the example of FIG. 11, the state determination device 1C includes score calculation units 18A to 18C, which respectively correspond to the explanatory variable calculation units 17A to 17C, as examples of the plurality of score calculation units 18. Hereinafter, when the explanatory variable calculation units 17A to 17C are collectively referred to, they will also be referred to as the explanatory variable calculation unit 17. Furthermore, hereinafter, when the score calculation units 18A to 18C are collectively referred to, they will also be referred to as the score calculation unit 18. The state determination device 1C also includes a score determination section 19, an identification section 70, and an explanatory variable selection section 71. The data acquisition unit 11 according to the present embodiment outputs sensor information to the identification unit 70 and the explanatory variable calculation unit 17, respectively.

The identification unit 70 acquires sensor information from the sensor group 2 and identifies the occupant from the sensor information. For example, the identification unit 70 may identify the occupant from an image taken by the camera 21. Further, when the sensor group 2 includes a microphone, the identification unit 70 may identify the occupant by recognizing the voice uttered by the occupant acquired by the microphone. The identification unit 70 outputs the passenger identification result to the explanatory variable selection unit 71.

The explanatory variable selection unit 71 reads explanatory variable data 33 corresponding to the occupant identified by the identification unit 70, and selects an explanatory variable to be used for determining the status of the occupant. Details of the method for selecting explanatory variables will be described later. The explanatory variable selection unit 71 instructs the explanatory variable calculation unit 17 corresponding to the explanatory variable used to determine the state of the occupant to calculate an explanatory variable.

Each of the explanatory variable calculation units 17A to 17C uses a different method to calculate an index value indicating whether the occupant is in a specific state based on the sensor information obtained from the sensor group 2. That is, in this embodiment, each of the explanatory variable calculation units 17A to 17C calculates an index value indicating whether or not the occupant is sleepy. Each of the index values calculated by the explanatory variable calculation units 17A to 17C is an explanatory variable used to ultimately obtain a target variable indicating "whether or not the occupant is in a particular state." Hereinafter, the index values calculated by each explanatory variable calculation unit 17 will also be referred to as explanatory variables.

The specific calculation method of each explanatory variable is not limited. For example, any of the explanatory variable calculation units 17A to 17C may calculate the sleepiness level as the explanatory variable, similarly to the first determination unit 12. Further, in the state determination device 1C, the number of explanatory variable calculation units 17 and score calculation units 18 is not particularly limited as long as it is plural. The explanatory variable calculation units 17A to 17C output the calculated explanatory variables to the score calculation units 18A to 18C, respectively.

Each of the score calculation units 18A to 18C calculates a score based on the explanatory variables input from the corresponding explanatory variable calculation units 17A to 17C. Specifically, when the explanatory variables are input, the score calculation units 18A to 18C refer to the explanatory variable data 33 and determine whether each explanatory variable is equal to or greater than the overall threshold and equal to or greater than the individual threshold. . When the index value is equal to or higher than the overall threshold and equal to or higher than the individual threshold, the score calculation units 18A to 18C each calculate the corresponding index value by substituting the index value into the score formula for each index value stored in the explanatory variable data 33. Calculate the score of the index value. The score calculation units 18A to 18C each output the calculated scores to the score determination unit 19. Note that the overall threshold, individual threshold, and score formula will be detailed later.

The score determination unit 19 calculates a total score that is the sum of each score, and determines whether the occupant is in a specific state based on the total score. In this embodiment, it can be said that the determination result of the score determination unit 19 is the final result of determining the condition of the occupant.

Specifically, the score determination unit 19 determines that the occupant is in a specific state when the total score is greater than or equal to a predetermined third threshold, and when the total score is less than the third predetermined threshold, the score determination unit 19 determines that the occupant is in a specific state. It is determined that the condition is not specific. That is, in the present embodiment, the score determination unit 19 determines that the occupant is sleepy when the total score is equal to or greater than the third threshold, and determines that the occupant is not sleepy when the total score is less than the third threshold. Note that the score determination unit 19 may normalize the value of the total score before determining the presence or absence of drowsiness. The specific value of the third threshold may be determined as appropriate depending on the value range of the explanatory variable. For example, the third threshold may be about 0.5 when the value of each explanatory variable is within the range of 0 to 1. The score determination section 19 outputs the determination result to the notification section 15.

The notification unit 15 according to the present embodiment causes the output device 5 to output a fourth notification when the score determination unit 19 determines that the occupant is in a specific state. The content of the fourth notification may be, for example, a warning to the occupant, similar to the first to third notifications.

(Explanatory variable data 33)
The explanatory variable data 33 is data in which the overall threshold value and the individual threshold value, which serve as determination criteria for each explanatory variable, are recorded. Further, the explanatory variable data 33 is data indicating whether each occupant is in a specific state and the strength of the correlation with each explanatory variable. The explanatory variable data 33 is created in advance for each occupant using the state determining device 1C or another device, and is stored in the storage device 3.

FIG. 12 is a diagram showing the data structure of the explanatory variable data 33. The explanatory variable data 33 is data in which an explanatory variable name is associated with an overall threshold value, an individual threshold value, and a score formula of the explanatory variable. One record of explanatory variable data 33 indicates explanatory variable data regarding one explanatory variable. In FIG. 12, the "explanatory variable" column stores information indicating explanatory variable names. The “overall threshold” column stores the overall threshold. The “personal threshold” column stores personal thresholds. “Score formula” stores information indicating the score formula. In the example of FIG. 12, the explanatory variable data 33 further includes information indicating the priority order of each explanatory variable.

(How to create explanatory variable data 33)
The explanatory variable data 33 is created based on sample data of individual occupants (hereinafter referred to as personal data) and sample data of multiple persons (hereinafter referred to as overall data). Note that the overall data may include personal data. Further, it is preferable that the number of people making up the entire data is large.

Both the personal data and the overall data include values of explanatory variables (i.e., index values) calculated in the same manner as each of the explanatory variable calculation units 17, and the probability of being determined to be "drowsy" based on the index values, This is data that corresponds to The probability of being judged as "drowsy" is determined by measuring whether the occupant was actually in a specific state for each index value included in the overall data or personal data, and then applying the measured data to a method such as logistic regression. It can be calculated by classifying values. For example, the personal data and the overall data may be data that associates the sleepiness level described in Embodiment 1 with the probability that the subject actually becomes sleepy at the sleepiness level.

FIG. 13 is a graph showing the overall threshold, individual threshold, and score formula when there is no correlation between a certain explanatory variable and a change to a specific state of the occupant. FIG. 14 is a graph showing the overall threshold, individual threshold, and score formula when there is a correlation between the same explanatory variables as in FIG. 13 and a change to a specific state of the occupant. The horizontal axes in FIGS. 13 and 14 indicate index values.

The vertical axes in FIGS. 13 and 14 indicate the probability of being determined to be "drowsy" based on the index value indicated by the horizontal axis. Further, the solid line bar graphs in FIGS. 13 and 14 indicate the distribution of personal data. On the other hand, the dotted bar graph in FIG. 13 shows the distribution of the entire data. Note that the graphs in FIGS. 13 and 14 are for visually explaining the method of calculating each value in the explanatory variable data 33, and there is no need for the state determination device 1C to actually create the illustrated graphs.

The "overall threshold" is the average value of the probability that the occupant is determined to be in a specific state (drowsy in this embodiment) in all samples of the overall data. The "individual threshold" is the average value of the probability that the occupant is determined to be in a specific state in all samples of personal data. The "score formula" is a mathematical formula that indicates the correlation between explanatory variables and the probability that the occupant is in a specific state. As illustrated, the score formula is, for example, a variable formula of a smoothed curve obtained by smoothing personal data. As will be described later, the value obtained by substituting the value of the explanatory variable into the score formula becomes the score of the explanatory variable. That is, the "score" in this embodiment is a numerical value calculated for each explanatory variable, and is a value indicating the possibility that the occupant is in a specific state.

As shown in Figure 13, even if an explanatory variable generally has a correlation with a specific state, if there is no correlation with a change to a specific state of a certain passenger, the value of the explanatory variable of personal data Even if this changes, the probability of being judged as "drowsy" does not change significantly. On the other hand, when the value of the explanatory variable of the overall data changes, the probability of being determined to be "drowsy" increases. Therefore, as shown in FIG. 13, the individual threshold is lower than the overall threshold.

On the other hand, as shown in Figure 14, in the case of an explanatory variable that has a correlation with a change in a certain state of a certain occupant, when the value of the explanatory variable in personal data changes, the probability of being judged as "drowsy" increases. do. If the explanatory variables of personal data have a higher degree of correlation than the explanatory variables of the overall data, the individual threshold becomes higher than the overall threshold, as shown in FIG.

In this way, the state determination device 1C or other device calculates the overall threshold, the individual threshold and the score formula for each occupant in advance for each explanatory variable, and records them in the explanatory variable data 33. Thereby, the explanatory variable calculation unit 17 (described later) of the state determination device 1C can calculate a score from the calculated values of the explanatory variables.

≪Processing flow≫
FIG. 15 is a flowchart showing the flow of processing in the state determining device 1C. In FIG. 15, the time when measurement of various information in the sensor group 2 is started is shown as the start of the process. When the camera 21 starts taking images and the pulse sensor 22 starts measuring the pulse (S21), the data acquisition unit 11 successively acquires various sensor information from the sensor group 2. The data acquisition unit 11 outputs the acquired sensor information to the identification unit 70 and each of the explanatory variable calculation units 17A to 17C. Note that the data acquisition unit 11 does not need to output all the sensor information to all the explanatory variable calculation units 17, respectively. The data acquisition unit 11 may output at least the sensor information necessary for calculating explanatory variables in each of the explanatory variable calculation units 17A to 17C to each of the explanatory variable calculation units 17A to 17C.

The identification unit 70 identifies the occupant based on the sensor information, and outputs the identification result to the explanatory variable selection unit 71. The explanatory variable selection unit 71 reads explanatory variable data 33 corresponding to the occupant identified by the identification unit 70, and selects an explanatory variable to be used for determining the status of the occupant (S22). The explanatory variable selection unit 71 instructs the explanatory variable calculation unit 17 corresponding to the explanatory variable used to determine the state of the occupant to calculate an explanatory variable.

Thereafter, each explanatory variable calculation unit 17 that receives instructions from the explanatory variable selection unit 71 and the corresponding score calculation unit 18 calculate each explanatory variable and score. Note that the order and timing of calculating each explanatory variable and calculating the score are not particularly limited as long as they are after S22.

First, each explanatory variable calculation unit 17 calculates an index value to be an explanatory variable (S23). Each of the explanatory variable calculation units 17 outputs the calculated explanatory variables to the corresponding score calculation unit 18. When each explanatory variable is input, the score calculation unit 18 refers to the explanatory variable data 33 and specifies the overall threshold and individual threshold of the input explanatory variable. The score calculation unit 18 each determines whether each explanatory variable is greater than or equal to the overall threshold value and greater than or equal to the individual threshold value corresponding to the explanatory variable (S24).

If the explanatory variable is equal to or greater than the overall threshold and equal to or greater than the individual threshold (YES at S24), the score calculation unit 18 that made the determination determines a score according to the value of the explanatory variable (S25). For example, if YES in S23, the score calculation unit 18 calculates the score by substituting the value of the explanatory variable into the score formula of the explanatory variable stored in the explanatory variable data 33.

On the other hand, if the explanatory variable is less than the overall threshold or at least the explanatory variable is less than the individual threshold (NO in S24), the score calculation unit 18 that made this determination sets the score of the explanatory variable to 0 ( S26).

In either case, each score calculation unit 18 outputs the calculated score to the score determination unit 19. When the calculation of all explanatory variables and the score calculation of all explanatory variables are completed, the score determination unit 19 calculates a total score by summing up the input scores (S27). The score determination unit 19 further determines whether the total score is equal to or greater than a third threshold (S28). If the total score is equal to or greater than the third threshold (YES in S28), the score determining unit 19 determines that the occupant is sleepy (S29). Upon receiving the determination result, the notification unit 15 causes the output device 5 to output a fourth notification (S30). On the other hand, if the total score is less than the third threshold (NO in S28), the score determination unit 19 determines that the occupant is not sleepy (S31). In this case, the notification unit 15 does not cause the output device 5 to output the fourth notification and ends the process.

≪Selection of explanatory variables≫
Finally, the selection of explanatory variables by the explanatory variable selection unit 71 will be explained. FIG. 16 is a diagram schematically showing the selection of explanatory variables used to determine the state of a certain occupant and the results of state determination using the explanatory variables. For convenience, in the example of FIG. 16, the occupant identified by the identification unit 70 is referred to as occupant A. Further, in FIG. 16, the third threshold value is assumed to be 0.5.

The first line in FIG. 16 shows the type of explanatory variable. In the example of FIG. 16, the state determination device 1C includes six explanatory variable calculation units 17 and six score calculation units 18 corresponding to six types of explanatory variables X1 to X6. The explanatory variable selection unit 71 selects an explanatory variable to be used for determining the state of the occupant A by referring to the explanatory variable data 33 of the occupant A. In the illustrated example, the explanatory variable selection unit 71 selects four explanatory variables X1 to X4 as explanatory variables to be used for determining the condition of the occupant, in order of priority stored in the explanatory variable data 33. .

The number of explanatory variables to be selected is not particularly limited. Furthermore, the method for selecting explanatory variables is not limited to the above method. For example, the explanatory variable selection unit 71 may use a predetermined number of explanatory variables in order of the highest personal threshold value in the explanatory variable data 33 of the occupant A to determine the condition of the occupant. Alternatively, the state determination device 1C refers to the score formula of each explanatory variable in the explanatory variable data 33 of the occupant A, and assigns a predetermined number of explanatory variables in descending order of the maximum value of the score formula to determine the state of the occupant. It may also be used.

The second line in FIG. 16 shows the comparison results between the value of each explanatory variable and the overall threshold and individual threshold. Moreover, the third line of FIG. 16 shows the score of each explanatory variable. Note that in FIG. 16, the "threshold" indicates the larger of the overall threshold and the individual threshold. That is, in FIG. 16, the case where the value of the explanatory variable is equal to or greater than the threshold value indicates that the value of the explanatory variable is equal to or greater than the overall threshold value and greater than or equal to the individual threshold value. In the illustrated example, for the explanatory variables X1 to X3, the values of the explanatory variables are greater than or equal to the threshold values, so each score calculation unit 18 respectively substitutes the values of the explanatory variables X1 to X3 into the score formula of each explanatory variable. and find the score. On the other hand, the value of the explanatory variable X4 is less than or equal to the threshold value. Therefore, the score calculation unit 18 corresponding to the explanatory variable X4 sets the score of the explanatory variable X4 to 0.

Thereby, if it is estimated from the value of the explanatory variable that the possibility that occupant A is feeling sleepy is low, the score of the explanatory variable can be set to 0, and it can be prevented from contributing to the total score. Therefore, the accuracy of state determination can be improved.

In the case of FIG. 16, the score determination unit 19 calculates the total score by summing up the scores derived from the explanatory variables X1, X2, and X3, as illustrated. Since the total score is equal to or greater than the third threshold of 0.5, it is determined that the occupant A is sleepy.

According to the process according to the present embodiment, it is possible to select explanatory variables that have a high correlation with changes to a specific state of the occupant, and to determine the final state determination result by summing the scores of the explanatory variables. . Therefore, the condition of the occupant can be determined more accurately.

In other words, according to the process according to the present embodiment, it is possible to avoid using explanatory variables that have a low correlation with a change in the occupant's specific state for state determination. Therefore, it is possible to accurately determine whether or not the occupant is in a specific state.

[Embodiment 4]
The state determination method according to Embodiment 1 or 2 and the state determination method according to Embodiment 3 may be used together. A fourth embodiment of the present disclosure will be described below. Note that members having the same functions as those described in any of Embodiments 1 to 3 will be denoted by the same reference numerals, and the description will not be repeated.

FIG. 17 is a block diagram showing the main configuration of the state determination system 400 according to this embodiment. The state determination system 400 has a configuration that combines the state determination system 100 according to the first embodiment and the state determination system 300 according to the third embodiment. Although the internal configuration of the server 4 is not shown in FIG. 17 to simplify the drawing, the server 4 has the same configuration as the server 4 described in the first and second embodiments.

The storage device 3 according to this embodiment stores user information 31, a status log 32, and explanatory variable data 33. Further, the state determining device 1D according to the present embodiment includes a plurality of second determining units 14 in place of the explanatory variable calculating unit 17 of the state determining device 1C. In the example of FIG. 17, the state determining device 1D includes three second determining sections 14A to 14C.

The processing procedure for determining the condition of the occupant in the condition determining device 1D will be described below. The second determining units 14A to 14C have the same configuration as the second determining unit 14 described in the first and second embodiments. The second determination units 14A to 14C each perform the second determination described in the first embodiment. Note that the start timing of the second determination is the same as in FIG. 4. When the second determination units 14A to 14C determine that the occupant is in a specific state, that is, “sleepy,” the second determination units 14A to 14C send the second index used for the second determination to the corresponding score calculation units 18A to 18C, respectively. Output the value. Note that the second determination units 14A to 14C do not need to output the determination results to the notification unit 15. Furthermore, the notification unit 15 does not need to output the second notification.

Each of the score calculation units 18A to 18C uses the input second index value as an explanatory variable, and calculates a score for each explanatory variable, similarly to the third embodiment. The score calculation units 18A to 18C each output the calculated scores to the score determination unit 19. The subsequent processing is similar to that of the state determining device 1C according to the third embodiment.

Note that when the state determination device 1D according to the present embodiment performs the second determination as described above, the explanatory variable data 33 does not need to include information indicating the overall threshold value and the individual threshold value. Moreover, in this case, the state determination device 1D executes only the second determination related to the calculation of a predetermined number of second index values starting from the one with the highest priority in the explanatory variable data 33 among the plurality of second determinations. It's okay. For example, in the state determination device 1D that can perform six types of second determinations, only the determinations of the top four priority determinations may be performed.

Further, the condition determining device 1D according to the present embodiment may determine the condition of the occupant using the following processing procedure. The second determination units 14A to 14C each include at least the second index value calculation unit 141 and the comparison determination unit 143 described in the first and second embodiments. The second determination units 14A to 14C each perform the second determination described in the first embodiment. However, the second determination units 14A to 14C do not determine the second threshold value shown in S162 to S166 in FIG. 6 and do not perform the comparison determination between the second index value and the second threshold value shown in S167.

The second determination units 14A to 14C refer to the explanatory variable data 33 and execute the process in S24 in FIG. 15 instead of S162 to S166 and S167 in FIG. That is, the second determination units 14A to 14C determine that the occupant is in a specific state, that is, "drowsy" when the calculated second index value is equal to or greater than the overall threshold and equal to or greater than the individual threshold. do. If it is determined that the state is a specific state, the second determination units 14A to 14C output the second index values as explanatory variables to the corresponding score calculation units 18A to 18C, respectively.

Each of the score calculation units 18A to 18C uses the input second index value as an explanatory variable, and calculates a score for each explanatory variable, similarly to the third embodiment. The score calculation units 18A to 18C each output the calculated scores to the score determination unit 19. The subsequent processing is similar to that of the state determining device 1C according to the third embodiment.

The state determination system 400 according to this embodiment may have a configuration that combines the state determination system 200 according to the second embodiment and the state determination system 300 according to the third embodiment. That is, the state determining device 1D may include the third determining section 16. The function of the third determination unit 16 is the same as that described in the second embodiment. In this case, the various processes in the second determination units 14A to 14C, score calculation units 18A to 18C, and score determination unit 19 and the third determination in the third determination unit 16 may be performed in parallel. .

[Modified example]
Part or all of the various processes in the state determination devices 1A to 1D according to each embodiment of the present disclosure may be executed by the server 4. For example, the state determination devices 1A to 1D according to the first embodiment include a data acquisition section 11, a communication section 13, and a notification section 15, and the processing performed by the other members of the state determination devices 1A to 1D is as follows. The server control unit 42 of the server 4 may execute it. In this case, the state determination devices 1A to 1D transmit the sensor information acquired by the data acquisition unit 11 to the server 4 via the communication unit 13. Further, the communication unit 13 of the state determination devices 1A to 1D receives from the server 4 the determination result of at least one of the first determination, the second determination, the third determination, and the state determination using the total score, The determination result is output to the notification section 15. As described in each of the above-described embodiments, the notification unit 15 determines whether or not to output various notifications from the output device 5 according to the determination result, and when outputting various notifications, selects one of the first to fourth notifications. is output to the output device 5.

Further, the condition determining devices 1A to 1D according to each embodiment of the present disclosure may be devices that determine various conditions of the occupant, not only whether the occupant is drowsy or not. For example, the state determining devices 1A to 1D may be devices that determine whether the occupant is in an inappropriate state for at least one of driving and communication. Here, "a state inappropriate for at least one of driving or communication" is, for example, an angry state or an absentminded state. Further, the various sensors included in the sensor group 2 may be sensors capable of acquiring information necessary for specifying the state to be determined by the state determining devices 1A to 1D. For example, if the state determining devices 1A to 1D are devices that determine whether or not the occupant is in an angry state, the sensor group 2 may include a camera 21 and a pressure sensor installed on the seat. good. The index value for identifying the state used for various determinations may be identified from the facial expression of the occupant in the image taken by the camera 21. Further, the index value may be specified from the movement of the occupant when seated, which is indicated by the measurement data of the pressure sensor.

[Example of implementation using software]
The control blocks of the status determination devices 1A, 1B, 1C, 1D and the server 4 may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software. good.

In the latter case, the state determination devices 1A, 1B, 1C, 1D, and the server 4 are equipped with a computer that executes instructions of a program that is software that implements each function. This computer includes, for example, one or more processors and a computer-readable recording medium that stores the above program. Then, in the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present disclosure. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, in addition to "non-temporary tangible media" such as ROM (Read Only Memory), tapes, disks, cards, semiconductor memories, programmable logic circuits, etc. can be used. Further, the computer may further include a RAM (Random Access Memory) for expanding the above program. Furthermore, the program may be supplied to the computer via any transmission medium (communication network, broadcast waves, etc.) that can transmit the program. Note that one aspect of the present disclosure can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The present disclosure is not limited to the embodiments described above, and various changes can be made within the scope of the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. are also included within the technical scope of the present disclosure.

100, 200, 300, 400 State determination system 1A, 1B, 1C, 1D State determination device 11 Data acquisition section 12 First determination section 13 Communication section 14 Second determination section 141 Second index value calculation section 142 Threshold calculation section 143 Comparison Determination unit 15 Notification unit 16 Third determination unit 17 Explanatory variable calculation unit 18 Score calculation unit 19 Score determination unit 70 Identification unit 71 Explanatory variable selection unit 2 Sensor group 21 Camera 22 Pulse sensor 3 Storage device 31 User information 32 Status log 33 Explanation Variable data 4 Server 41 Server communication unit 42 Server control unit 43 Server storage unit 431 Collection data 5 Output device 6 Mobile object

Claims (10)

an acquisition unit that continuously acquires sensor information regarding an occupant of the mobile body from one or more sensor devices arranged in an internal space of the mobile body;
an index value calculation unit that calculates a state index value indicating the state of the occupant based on at least part of the sensor information;
a receiving unit that receives collected data from the management device;
a state threshold calculation unit that calculates, based on the collected data, a state threshold that is a threshold for determining whether the state of the occupant indicated by the state index value is a specific state;
a state determining unit that determines that the state of the occupant is in the specific state when the state index value is equal to or greater than the state threshold;
The collected data associates the condition index value of the occupant of the other moving object calculated in the other moving object with the condition of the occupant of the other moving object determined in the other moving object. Contains multiple data ,
The sensor information is data indicating a biological heart rate,
The state determination device determines whether the state of the occupant is a specific state based on at least a part of the sensor information before the index value calculation unit calculates the state index value. further comprising a sub-state determining section;
The state determining device, wherein the index value calculation unit specifies the state index value using the sensor information for a predetermined first period . an average determination unit that determines whether the occupant is in a specific state based on the state index value or a value of an intermediate result when calculating the state index value from the sensor information;
The average determination unit determines whether the condition index value or the value of the intermediate result is equal to or greater than the average value of the condition index values of the occupant in a past second period, or The condition determining device according to claim 1 , wherein the condition determining device determines that the occupant is in a specific condition when the intermediate result value is equal to or greater than an average value. The state threshold calculation unit includes:
A value that is greater than or equal to the median of the state index values calculated for the other moving objects, and
According to claim 1 or 2 , a value for which half or more of the plurality of data indicates that the occupant is in a specific state is set as the state index value for the moving object. status determination device. The state determining device includes:
an explanatory variable selection unit that selects, as an explanatory variable, a state index value that has a high correlation with a change in the identified specific state of the occupant;
a plurality of index value calculation units that calculate the state index values selected by the explanatory variable selection unit;
a score calculation unit that calculates a score indicating the possibility that the occupant is in a specific state with respect to the state index values calculated by each of the plurality of index value calculation units;
4. The vehicle according to claim 1, further comprising: a score determining unit that determines that the occupant is in a specific state when the total score obtained by summing the scores is equal to or higher than a predetermined score threshold. The state determination device according to item 1. an acquisition unit that continuously acquires sensor information regarding an occupant of the mobile body from one or more sensor devices arranged in an internal space of the mobile body;
an index value calculation unit that calculates a state index value indicating the state of the occupant based on at least part of the sensor information;
a receiving unit that receives collected data from the management device;
a state threshold calculation unit that calculates, based on the collected data, a state threshold that is a threshold for determining whether the state of the occupant indicated by the state index value is a specific state;
a state determining unit that determines that the state of the occupant is in the specific state when the state index value is equal to or greater than the state threshold;
The collected data associates the condition index value of the occupant of the other moving object calculated in the other moving object with the condition of the occupant of the other moving object determined in the other moving object. Contains multiple data,
The state determining device includes:
an explanatory variable selection unit that selects, as an explanatory variable, a state index value that has a high correlation with a change in the identified specific state of the occupant;
a plurality of index value calculation units that calculate the state index values selected by the explanatory variable selection unit;
a score calculation unit that calculates a score indicating the possibility that the occupant is in a specific state with respect to the state index values calculated by each of the plurality of index value calculation units;
A state determining device further comprising: a score determining unit that determines that the occupant is in a specific state when a total score obtained by adding up the scores is equal to or higher than a predetermined score threshold. 4 . The score calculation unit calculates the score based on a formula that indicates a correlation between the state index value of the identified passenger and the probability of being in the specific state. 4 . or the state determination device according to 5 . The vehicle according to any one of claims 1 to 6 , further comprising a notification unit that notifies the occupant of a warning via an output device when the condition of the occupant is determined to be a specific condition. The condition determination device described. A state determination device according to any one of claims 1 to 7 ,
The sensor device;
A state determination system comprising: the management device. A control method executed by a state determination device including a processor, the method comprising:
The processor includes:
Continuously acquiring sensor information regarding the occupant of the mobile body from one or more sensor devices arranged in the interior space of the mobile body ,
Calculating a state index value indicating the state of the occupant based on at least a part of the sensor information ,
Receive collected data from the management device,
Based on the collected data, calculate a state threshold that is a threshold for determining whether the state of the occupant indicated by the state index value is a specific state ;
If the state index value is equal to or greater than the state threshold, determining that the state of the occupant is in the specific state ,
The collected data associates the condition index value of the occupant of the other moving object calculated in the other moving object with the condition of the occupant of the other moving object determined in the other moving object. Contains multiple data,
The sensor information is data indicating a biological heart rate,
The processor includes:
Before calculating the state index value, determine whether the state of the occupant is in a specific state based on at least a part of the sensor information,
A control method that specifies the state index value using the sensor information for a predetermined first period . A control method executed by a state determination device including a processor, the method comprising:
The processor includes:
Continuously acquiring sensor information regarding the occupant of the mobile body from one or more sensor devices arranged in the interior space of the mobile body,
Calculating a state index value indicating the state of the occupant based on at least a part of the sensor information,
Receive collected data from the management device,
Based on the collected data, calculate a state threshold that is a threshold for determining whether the state of the occupant indicated by the state index value is a specific state;
If the state index value is equal to or greater than the state threshold, determining that the state of the occupant is in the specific state,
The collected data associates the condition index value of the occupant of the other moving object calculated in the other moving object with the condition of the occupant of the other moving object determined in the other moving object. Contains multiple data,
The processor includes:
Selecting as an explanatory variable a state index value that has a high correlation with a change to the specific state of the identified passenger;
calculating a plurality of the selected state index values;
Calculating a score indicating the possibility that the occupant is in a specific state for the plurality of calculated state index values,
A control method in which it is determined that the occupant is in a specific state when a total score obtained by summing the scores is equal to or greater than a predetermined score threshold.

Images