JP2022130174A - Dozing detection device, dozing detection method, and computer program - Google Patents

Abstract

To provide a dozing detection device which detects dozing accurately.SOLUTION: A dozing detection device 100 is a detection device for detecting dozing and comprises an arithmetic unit 1 which executes detection processing for detecting dozing. The detection processing includes obtaining an index value for detecting dozing from an RRI (R-R Interval) generated from an electrocardiographic signal of a subject, measuring a duration time in which the index value exceeds a first threshold, and detecting dozing of the subject when the duration time exceeds a second threshold.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a doze detection device, a detection method, and a computer program.

As a technique for detecting dozing off of a subject, for example, Japanese Patent Application Laid-Open No. 2015-226696 (hereinafter referred to as Patent Document 1) proposes a method of using the electrocardiogram of the subject.

JP 2015-226696 A

Such dozing detection is also assumed to be used when detecting dozing of a driver of a vehicle or the like. Therefore, further improvement in detection accuracy is desired.

A doze detection device according to an embodiment is a detection device that detects doze, and includes an arithmetic device that executes detection processing for detecting doze, wherein the detection processing is generated from an electrocardiographic signal of a subject. An index value for detecting dozing off is obtained from RRI (R-R Interval), the duration for which the index value exceeds a first threshold is measured, and when the duration exceeds a second threshold, dozing of the subject is detected. including.

A detection method according to an embodiment is a method for detecting dozing, in which an index value for dozing detection is obtained from an RRI generated from an electrocardiographic signal of a subject, and the index value exceeds a first threshold. Timing the duration and detecting the subject falling asleep when the duration exceeds a second threshold.

A computer program according to an embodiment is a computer program for causing a computer to execute detection processing for detecting dozing, wherein the detection processing is based on an RRI generated from an electrocardiographic signal of a subject to detect dozing. obtaining an index value for, timing the duration for which the index value exceeds a first threshold, and detecting drowsiness of the subject when the duration exceeds a second threshold.

Further details are described as embodiments below.

FIG. 1 is a diagram showing an outline of the configuration of a dozing detection device (hereinafter referred to as a detection device) according to an embodiment. (a) of FIG. 2 is a diagram showing an example of an electrocardiographic signal, and (b) is a diagram showing R-wave data corresponding to the electrocardiographic signal of (a). FIG. 3 is a block diagram showing an example of the configuration of the arithmetic unit of the detection device. FIG. 4 is a diagram for explaining the Poincaré index, the T2 statistic, and the Q statistic. FIG. 5 is a diagram for explaining canonical correlation analysis of data sets. FIG. 6 is a diagram for explaining a specific example of a method for dynamizing a data set. FIG. 7 is a flow chart showing an example of a doze detection method in the detection device according to the first embodiment. FIG. 8 is a flow chart showing an example of determination processing in step S300 of the flow chart of FIG. FIG. 9 shows test results for verifying which of the Q statistic and the T2 statistic is used as the index value for better detection accuracy in the detection method according to the first embodiment by the inventors. It is the figure which represented. FIG. 10 shows the ROC (receiver operating Characteristic) curve. FIG. 11 is an ROC curve obtained when only Poincare index values are used as index values used as data sets to be input to the discriminator in the detection method according to the first embodiment by the inventors. . FIG. 12 is obtained by canonical correlation analysis of the values of the HRV index and the Poincare index as index values used as data sets to be input to the discriminator in the detection method according to the first embodiment by the inventors. ROC curves obtained when using canonical variables. FIG. 13 is a time series obtained by dynamizing the values of the HRV index and the Poincare index as index values used as data sets to be input to the discriminator in the detection method according to the first embodiment by the inventors. Fig. 10 is a ROC curve obtained when using time-series data with variation; FIG. 14 is obtained by canonical correlation analysis of the values of the HRV index and the Poincare index as index values used as data sets to be input to the discriminator in the detection method according to the first embodiment by the inventors. 10 is a ROC curve obtained when using time-series data having time-series changes obtained by dynamizing the canonical variables. FIG. 15 is a diagram showing the results of a test conducted by the inventors to verify how many beats of past data should be incorporated in dynamization in the detection method according to the first embodiment. FIG. 16 is a flow chart showing an example of a method of detecting dozing in the detection device according to the second embodiment. FIG. 17 is a diagram showing the results of an evaluation experiment conducted by the inventors regarding the detection method according to the second embodiment for electrocardiogram signals including a state in which the subject is dozing. FIG. 18 is a diagram showing the results of an evaluation experiment conducted by the inventors regarding the detection method according to the second embodiment for electrocardiographic signals in which the subject is always awake. FIG. 19 is a diagram showing the results of an evaluation test conducted by the inventors in the case where the configuration of the autoencoder used in the detection method according to the second embodiment is LSTM and self-attention.

<1. Overview of Doze Detection Device, Detection Method, and Computer Program>

(1) A doze detection device according to an embodiment is a detection device that detects doze, and includes an arithmetic device that executes detection processing for detecting doze, and the detection processing is performed from an electrocardiographic signal of a subject. An index value for dozing detection is obtained from the generated RRI (R-R Interval), the duration of the index value exceeding the first threshold is measured, and when the duration exceeds the second threshold, the subject's dozing is detected. to do, including

Index values for doze detection are, for example, the following errors and statistics. The first threshold is a value at which the index value can be determined as dozing off. The possibility of the subject falling asleep is determined by determining that the index value exceeds the first threshold.

In order to obtain an index value for doze detection, an autoencoder, a discriminator using MSPC (Multivariate Statistical Process Control), and a deep layer using LSTM (long short-term memory) A learning model, a deep learning model using self-attention, and the like can be used.

The second threshold is a duration during which the duration of a state in which it is determined that there is a possibility of dozing can be determined as dozing, and is, for example, about 10 seconds. About 10 seconds can be adopted as the second threshold, according to an evaluation experiment conducted by the inventors, which will be described later, when the second threshold is set to 10 seconds, the subject falls asleep, and after falling asleep It has been verified that wakefulness is accurately detected (FIGS. 17 and 18). As a result, it is possible to detect dozing with higher accuracy than determining dozing based on the index value exceeding the first threshold. This was also verified by a verification test by the inventors, which will be described later.

(2) Preferably, obtaining an index value for detecting doze is inputting a value obtained from the RRI to a model that uses the target data as an input value and outputs a value used to detect an abnormality in the target data. values and using the output values from the model to obtain the index values. The model is a computational model, for example a machine-learned model. Specifically, it is an autoencoder, a deep learning model using LSTM, a deep learning model using self-attention, and the like. Another example of the model is a model that performs statistical processing on input values, specifically MSPC. By using the model, the output from the model provides the values needed to calculate the index value.

(3) Preferably, the model is an autoencoder, and the index value is obtained by providing the autoencoder with RRI generated from the subject's electrocardiographic signal as input data and reconstructing the input data. It includes acquiring certain output data and calculating the error of the output data from the RRI generated from the subject's electrocardiographic signal as an index value. This makes it possible to easily detect doze with high accuracy.

(4) Preferably, the autoencoder is trained by providing the RRI generated from the electrocardiographic signal during wakefulness as input data for learning. In this case, the first threshold is different from the input data obtained from the RRI generated from the electrocardiographic signal when the input data is awake, that is, the input data obtained from the RRI generated from the electrocardiographic signal including the doze. By setting the value to a certain extent larger than the input data, the possibility of dozing off can be determined by comparing the error with the first threshold.

(5) Preferably, the model is an MSPC-based discriminator, and obtaining the index value includes generating a heartbeat-related index from an RRI generated from an electrocardiographic signal of the subject, and a plurality of the indices Inputting a data set containing the variable HRV to the discriminator and obtaining a statistic as an index value. The heartbeat index is, for example, a heart rate variability (HRV) index.

The statistic is, for example, at least one of the Q statistic and the T2 statistic. The Q statistic is represented by Equation (5) in FIG. 4, and the T2 statistic is represented by Equation (4) in FIG. In this case, the first threshold is a value at which the Q statistic and/or the T2 statistic can be determined to be dozing, and is, for example, a control limit value. A control limit value is, for example, a statistic obtained by inputting a data set generated only from RRI obtained from an awake person into a discriminator. Thereby, the possibility of falling asleep can be determined by comparing the statistics with the first threshold.

(6) Preferably, the plurality of variables HRV includes at least one variable included in the Poincaré index obtained from the RRI generated from the electrocardiographic signal of the subject. A verification test by the inventors has verified that this enables more accurate detection of dozing.

(7) Preferably, the dataset includes a plurality of canonical variables among the canonical variables obtained by canonical correlation analysis of a plurality of variables HRV. A verification test by the inventors has verified that this enables more accurate detection of dozing.

(8) Preferably, the data set includes data having time-series changes generated from heartbeat indices obtained from each of a plurality of heartbeats of the electrocardiographic signal of the subject. A verification test by the inventors has verified that this enables more accurate detection of dozing.

(9) Preferably, the data set includes at least one of the variables included in the Poincaré index obtained from the RRI generated from the electrocardiographic signal of the subject, a plurality of variables HRV For each of the canonical variables obtained by canonical correlation analysis of a plurality of variables HRV, the subject is generated from a heartbeat-related index obtained from each of a plurality of heartbeats of the electrocardiographic signal. A verification test by the inventors has verified that this enables more accurate detection of dozing.

(10) Preferably, obtaining the index value for doze detection is performed by an autoencoder, a discriminator using MSPC, a deep learning model using LSTM, and a deep learning model using self-attention. including using at least one of

(11) A detection method according to an embodiment is a method for detecting dozing, in which an index value for dozing detection is obtained from an RRI generated from an electrocardiographic signal of a subject, and the index value is a first threshold value. and detecting drowsiness of the subject when the duration exceeds a second threshold. This realizes the detection device described in (1) to (10).

(12) A computer program according to an embodiment is a computer program for causing a computer to execute detection processing for detecting dozing, wherein the detection processing is based on an RRI generated from an electrocardiographic signal of a subject, Obtaining an index value for doze detection, timing a duration for which the index value exceeds a first threshold, and detecting doze of the subject when the duration exceeds a second threshold. This allows the computer to function as the detection device described in (1) to (10).

<2. Example of doze detection device, detection method, and computer program>

[First embodiment]

FIG. 1 is a diagram showing an outline of the configuration of a doze detection device (hereinafter referred to as a detection device) according to this embodiment. With reference to the drawing, the detection device 100 includes an arithmetic device 1 and a heart rate measuring device 2 .

The detection device 100 is a detection device that detects dozing off of the subject based on the heartbeat of the subject. A subject is, for example, a driver of a vehicle. A vehicle is a bus, a taxi, a passenger car, a train, or the like. The arithmetic device 1 is, for example, a device that can be mounted on such a vehicle, a device that can be held by the subject, and a server that can communicate with these devices via communication such as the Internet. is one.

Detecting dozing includes detecting being awake. That is, it is possible to detect at least one of the two states of the subject, i.e., a dozing state and an awake state. Preferably, the detection device 100 detects both the dozing state and wakefulness state of the subject.

The computing device 1 and the heart rate measuring device 2 can communicate with each other. The communication may be wireless communication, as an example. Wireless communication includes, for example, short-range wireless communication such as Bluetooth (registered trademark), communication via the Internet, and the like.

The heart rate measuring device 2 is a small and lightweight wearable device that is attached to the subject's P body and measures the heart rate of the subject. A plurality of (three in FIG. 1) electrodes 21 attached to the body surface of the subject P are connected to the heart rate measuring device 2 . The three electrodes 21 are, for example, a positive electrode, a negative electrode and a ground electrode.

As a wearable terminal functioning as the heart rate measuring device 2, for example, a smart watch having a heart rate measuring function can be given. Note that the wearable terminal itself may function as the arithmetic device 1 and the heart rate measuring device 2 .

FIG. 2(a) is a diagram showing an example of an electrocardiographic signal. In FIG. 2A, the vertical axis indicates potential and the horizontal axis indicates time. When the heart rate is measured using the electrodes 21, potential changes consisting of P to T waves as shown in FIG. 2(a) appear periodically. The peak with the highest potential in the potential change in a unit cycle is called an R wave, and the heart beats at the timing of the R wave. The heartbeat measuring device 2 transmits R-wave data representing R-waves to the computing device 1 .

FIG. 2(b) shows R-wave data corresponding to the electrocardiographic signal of FIG. 2(a). As shown in (b) of FIG. 2, in the R-wave data, as an example, the period corresponding to the R-wave in the electrocardiographic signal (the period during which the signal intensity I exceeds a predetermined intensity threshold value Ith) is set to "1". , and other periods are data representing a rectangular pulse train set to "0".

The computing device 1 functions as a detection device that detects that the subject is in a dozing state based on the R-wave data transmitted from the heart rate measuring device 2 . Arithmetic device 1 is, for example, a communication terminal such as a smart phone or a personal computer. Alternatively, the arithmetic device 1 is a device that is installed in a device operated by a subject, such as an in-vehicle device, or that can be installed in an in-vehicle device or the like.

FIG. 3 is a block diagram showing an example of the configuration of the computing device 1. As shown in FIG. Referring to FIG. 3, arithmetic device 1 includes processing unit 10 . The processing unit 10 is composed of a dedicated microcomputer or the like. The arithmetic device 1 also has a memory 12 . The memory 12 stores a program 121 for performing calculations in the processing unit 10 . The program 121 has program code for processing in the arithmetic device 1 . The program 121 is read and executed by the processing unit 10 connected to the memory in which the program 121 is stored.

The program 121 may be provided as a program product recorded on a non-transitory computer-readable recording medium. Program 121 may be provided by download over a network. Also, the program 121 may be provided as an application program, or part or all of it may be included in the operating system (OS) of the computer.

In the following description, it is assumed that the processing in the arithmetic device 1 is implemented by the program 121, but as another example, it may be implemented by circuit elements or other hardware.

The computing device 1 has a communication section 13 that communicates with the heart rate measuring device 2 . The communication unit 13 receives, for example, a radio signal indicating an R wave transmitted from the heart rate measuring device 2 and inputs it to the processing unit 10 . Alternatively, the communication unit 13 may be a reading device that accesses a recording medium on which data representing the R wave is recorded and reads the data from the recording medium. That is, acquiring data includes receiving data through communication and reading data from a recording medium.

The processing unit 10 executes a detection process for detecting dozing by executing a program 121 stored in the memory 12 . The sensing process includes RRI calculation process 101 . The RRI calculation process 101 includes calculating an RRI (R-R Interval), which is an interval between R waves, from R wave data as shown in FIG. 2B input from the communication unit 13 and obtaining RRI data. . RRI data is time-series data of RRI variables.

As another example, the heartbeat measuring device 2 may detect RRI from R-wave data and output heartbeat data including the RRI data. The communication unit 13 receives heartbeat data output from the heartbeat measuring device 2 . In this case, the processing unit 10 executes processing for extracting RRI data included in the heartbeat data instead of the RRI calculation processing 101 . In other words, since the heart rate measuring device 2 outputs heart rate data including RRI data, the calculation device 1 does not need to perform the process of calculating the RRI. Also, the data transmitted from the heart rate measuring device 2 becomes RRI data having a size smaller than that of the radio signal indicating the R wave, which facilitates data transmission.

The detection processing includes index value calculation processing 102 . The index value calculation process 102 includes calculating an index value for doze detection using RRI. Specifically, the index value calculation process 102 includes using a model 105 for calculation, inputting a value obtained from the RRI into the model 105 , and obtaining an output value from the model 105 . The model 105 is a model that takes target data as an input value and outputs a value that is used to detect an abnormality in the target data.

Obtaining an index value for doze detection includes, as an example, generating a heart rate index from the RRI. A heartbeat-related index includes a heart rate variability (HRV) index (hereinafter referred to as an HRV index).

HRV indices include the following 10 indices.
1) meanNN: mean value of RRI 2) SDNN: standard deviation of RRI 3) Total Power: variance of RRI 4) RMSSD: root mean square of difference between adjacent RRIs 5) NN50: difference between adjacent RRIs of 50 ms Number of times exceeded 6) LF: power spectrum of low frequency (0.04 to 0.15 Hz) of PSD (Power Spectrum Density: power spectral density of RRI data) 7) HF: high frequency of PSD (0.15 to 0.40 Hz) 8) LF/HF: Ratio of LF to HF 9) LFnu: Corrected LF defined by LF/(LF+HF)
10) HFnu: corrected HF defined as HF/(LF+HF)

RMSSD has a large value when the RRI fluctuates a lot. RMSSD is used as an index of autonomic nervous system activity. A typical RRI variation is less than 50 milliseconds. Therefore, NN50 is an indicator of severe RRI fluctuations. That is, a large NN50 indicates a large variation in RRI.

LF is mainly used as an index of activity of the sympathetic nervous system, and HF is mainly used as an index of activity of the parasympathetic nervous system. Therefore, LF/HF represents the ratio of activity between the sympathetic nervous system and the parasympathetic nervous system, and a larger value indicates that the sympathetic nervous system is dominant, and a smaller value indicates that the parasympathetic nervous system is dominant. LFnu is an index value emphasizing changes in sensory nerve activity, and HFnu is an index value emphasizing changes in parasympathetic nerve activity.

Obtaining an index value for doze detection may further include generating a Poincaré index. FIG. 4 is a diagram for explaining the Poincaré index, plotting the RRI on the horizontal axis and the RRI after one beat on the vertical axis. The Poincaré index includes the lower three indices represented by ellipse E in FIG. All three indices represent the magnitude of fluctuations in RRI.
1) SD1: standard deviation of minor axis A2 of ellipse E 2) SD2: standard deviation of major axis A1 of ellipse E 3) SD1/SD2: ratio of SD1 to SD2

Obtaining an index value for dozing detection involves inputting a data set in which a plurality of indexes out of the 10 indexes above the HRV index are arranged in time series as variables into the model 105, and obtaining an output value from the model 105. Including. In the detection device 100 according to the first embodiment, the model 105 is an arithmetic model that performs statistical processing on input values, and as an example, MSPC (Multivariate Statistical Process Control) is used. is a discriminator. That is, in the first embodiment, to obtain an index value for doze detection, a classifier is applied to a data set in which a plurality of indices out of the top ten indices of the HRV index are arranged in time series as variables. Including applying.

MSPC is a statistical technique based on principal component analysis. Principal component analysis is a multivariate analysis aimed at feature extraction and dimensionality reduction of data, and creates new synthetic variables called principal components by linear combination of variables in order to capture correlations between variables. It is.

In MSPC, the first principal component is set in the direction that best represents the data, and the first principal component is set in the direction that best represents the fluctuation of the data that cannot be represented by the first principal component in the space orthogonal to the first principal component. Principal components are set one after another by the procedure of setting two principal components. The direction that best expresses the data is the direction in which the variance of the principal component score is maximized. . In the example of FIG. 4, the direction of the major axis A1 is the first principal component direction, and the direction of the minor axis A2 is the second principal component direction.

Here, using the number of variables P and the number of samples N, the data matrix X is defined as shown in Equation (1) in FIG. Each variable is assumed to be standardized. Also, using the orthogonal matrices U and V and the diagonal matrix S in which the singular values s and r are arranged in descending order on the diagonal elements, the singular value decomposition of the data matrix X is performed as shown in equation (2) in FIG. Define.

At this time, using the number R of principal components to be adopted, the r-th principal component is given by the r-th column vr of the loading matrix V _R , and the r-th principal component tr is given by the r-th column vr of the orthogonal matrix U _R is obtained by the formula (3) in FIG.

In the first embodiment, the model 105 outputs at least one of the Q statistic and the T2 statistic according to MSPC when a data set arranged in time series is input. In the index value calculation process 102, the value output by the model 105 is used as the index value for doze detection. The Q statistic and the T2 statistic are indices for evaluating the degree of deviation from normal values of variables.

The T2 statistic represents the distance from the mean to each sample in the principal component space obtained by compressing the original variables. That is, the T2 statistic represents the deviation of the input data set from the average. Specifically, the T2 statistic is represented by equation (4) in FIG. 4 using the r-th principal component tr represented by equation (3) in FIG. 4 and the standard deviation σtr of the r-th principal component score tr. be done. The T2 statistic corresponds to the Mahalanobis distance. That is, it can be said that the smaller the T2 statistic, the closer the sample is to the average of the model building data.

The Q statistic is an index that evaluates the deviation from the correlation between the variables of the input data set. The Q statistic can detect abnormal correlations between variables. The Q statistic is calculated from the residual and is expressed in detail by Equation (5) in FIG. The Q statistic is also called the squared prediction error and is taken to represent the portion of the data that cannot be represented by the model. The MSPC simultaneously monitors the T2 statistic and the Q statistic, and determines that it is abnormal when either one exceeds the control limit.

The data set input to the model 105 may be a data set in which the values of each of the 10 indices are arranged in chronological order, or a plurality of some of the indices may be used as elements. Values may be arranged in chronological order. Preferably, the data set input to the model 105 further comprises at least one of the three indicators above the Poincaré indicator. For example, the elements of the data set to be input may be values of a total of 13 indices, ie, the top 10 HRV indices and the top 3 Poincare indices, arranged in chronological order. It may also be a normalized data set.

Preferably, in obtaining index values for doze detection, the processing unit 10 performs canonical correlation analysis on the data set, and inputs the obtained data set consisting of canonical variables to the discriminator. Canonical correlation analysis is a technique for obtaining correlations between linearly transformed values for each of a plurality of data, and performing canonical correlation analysis includes multiplying a data set by sample canonical coefficients.

A sample canonical coefficient is a coefficient that expresses individual differences inherent in a data set as weights. Specifically, the sample canonical coefficients are obtained by canonical correlation analysis of the matrix (X) shown in FIG. 5 and its one-hot matrix (Y). Matrix (X) vertically concatenates data sets generated using RRIs obtained during wakefulness for each of N subjects numbered n (n=1, 2, 3, . . . , n). matrix. A one-hot matrix refers to a vector in which only one of the elements is 1 and the other elements are 0s. Specifically, it is obtained by concatenating N vectors in which the time-series data of the n-th person is 1 and the others are 0.

The processing unit 10 extracts at least some of the obtained canonical variables as a new data set and inputs it to the model 105 . For example, the first, tenth, eleventh, and twelfth variables among the obtained canonical variables are taken out as a new data set and input to the model 105 . At this time, the new data set preferably includes variables obtained from at least one of the three Poincare indexes.

Preferably, in obtaining the index value for doze detection, the processing unit 10 dynamizes the data set and inputs the obtained data set of time-series data having time-series changes to the model 105 . The data set to be dynamized may be a data set consisting of canonical variables obtained by canonical correlation analysis, or may be a data set consisting of variables before canonical correlation analysis.

Dynamizing is generating data with time-series variation using index values for a plurality of heartbeats obtained from each of a plurality of heartbeats of the subject's electrocardiographic signal for each element of the new data set. point to Specifically, dynamization may be data with time-series changes by adding past data as elements.

A specific example of a method for dynamizing the first data set D1 will be described with reference to FIG. The first data set D1 is time-series data whose elements are the values of heartbeats 1 to 5 of four variables, ie, the first variable, the tenth variable, the eleventh variable, and the twelfth variable.

In the dynamization of the first data set D1, as an example, the second data set D2 corresponding to time-series data one beat after the first data set D1 is used. The second data set D2 is the first data set D1 of each of the same four variables as the first data set D1, the first variable, the tenth variable, the eleventh variable, and the twelfth variable. This is time-series data whose elements are values from heartbeats 2 to 5 one beat after the heartbeat.

In dynamization, the third data set D3 is generated by concatenating the first data set D1 with the second data set D2. The third data set D3 is the values of the first variable, the tenth variable, the eleventh variable, and the twelfth variable in the heartbeat of the first data set D1, and the first variable in the heartbeat , 10th variable, 11th variable, and 12th variable, and the values of heart beats 2 to 5 after one beat, respectively.

In the dynamization of the first data set D1, it is also possible to combine with the first data set D1 a data set of beats after the first beat of the second data set D2. The later beat may be, for example, two beats later. In the example of FIG. 6, when the data sets after two beats are combined, the generated data set becomes time-series data consisting of 12 variables and 48 elements.

In the index value calculation process 102, by inputting a data set having time-series changes obtained by dynamization or a normalized version of the data set to the model 105, the Q statistic and A T2 statistic is obtained and at least one of them is used as an index value.

The detection process includes determination process 103 . Determination processing 103 compares a plurality of index values calculated from continuous heartbeats by index value calculation processing 102 with a first threshold, and determines duration DT1 exceeding the first threshold and duration DT1 remaining below the first threshold. Including timing the time DT2. The index value is at least one of a Q statistic and a T2 statistic obtained by inputting the data set to the discriminator.

The first threshold is a value at which the Q statistic and/or the T2 statistic can be determined to be dozing, and is, for example, the control limit value Q0 and/or the control limit value T0. The control limit value Q0 is, for example, a Q statistic obtained by inputting a data set generated from RRIs obtained from awake persons only and not including RRIs during doze into the model 105. be. As an example, the control limit value T0 is a T2 statistic obtained by inputting a data set generated from RRIs obtained from awake people only and not including RRIs during dozing into the model 105. be. As another example, the control limit value may be a value stored in advance as the Q statistic and/or the T2 statistic during wakefulness.

Determination processing 103 compares the index value with a first threshold value to determine whether the index value obtained from the electrocardiographic signal of the subject deviates from the index value obtained from the electrocardiographic signal during wakefulness. including determining whether That is, here, it is determined that the subject is not in an awake state, that is, is in a dozing state when the index value exceeds the first threshold. In a state in which the index value is below the first threshold, it is determined that the subject is in an awake state, that is, in a state of not dozing off.

Here, the electrocardiogram may be mixed with noise due to factors such as premature ventricular contraction (PVC), which is a common arrhythmia that can occur even in a healthy person, and body movements of the subject. Therefore, an accurate RRI may not be obtained. In such a case, even if the index value exceeds the first threshold, it may not necessarily mean that the user is dozing off. Therefore, more accurate doze detection is desired.

In addition, when the index value based on the obtained RRI falls below the first threshold value, it is determined that the user is awake, but in reality, the user may not be completely awake from a doze. . Therefore, detection of awakening with higher accuracy is also desired. In other words, more accurate doze detection is desired.

In this regard, in the determination processing 103, the duration DT1 exceeding the control limit value is measured for at least one of the Q statistic and the T2 statistic obtained as the index value. Then, determination processing 103 includes comparing the obtained duration DT1 with a second threshold, and determining that the user is dozing when the duration DT1 exceeds the second threshold. Also, for at least one of the Q statistic and the T2 statistic, the duration DT2 below the control limit value is measured. Then, the determination processing 103 includes comparing the obtained duration DT2 with a second threshold, and determining awakening when the duration DT2 exceeds the second threshold.

The second threshold is the duration of the state determined to have the possibility of dozing, the duration of the state determined to be dozing, and the duration of the state determined to have the possibility of awakening determined to be the wakeful state. It is a threshold t0 of possible duration, which is, for example, about 10 seconds. Note that different thresholds may be used for determining dozing and for determining wakefulness.

That is, in the detection device 100, the state in which the Q statistic and/or the T2 statistic, which are the index values, exceeds the control limit value continues for the threshold t0 or more, which is the duration at which it can be determined that the subject is dozing. detect. As a result, dozing can be detected with higher accuracy than determining dozing when the Q statistic and/or the T2 statistic, which are index values, exceed the control limit value.

Further, in the detection apparatus 100, when the state in which the Q statistic and/or the T2 statistic, which are the index values, is below the control limit value continues for a duration equal to or longer than the threshold value t0 that can be determined as an arousal state, the arousal state of the subject to detect. As a result, the arousal state is detected with higher accuracy than when the awake state is determined when the Q statistic and/or the T2 statistic, which are index values, are below the control limit value, in other words, dozing is detected with high accuracy. be able to.

Sensing processing may include output processing 104 . Output processing 104 includes outputting the detection result. The output of the detection result may be one or more of audio output, display output, lighting output, vibration, transmission to another device, and the like. Output processing 104 includes transmitting a control signal instructing output to an output device (not shown).

The detection result to be output may be a detection result indicating that the user is dozing off, or may be a detection result indicating that the user is not dozing off, that is, in an awake state. This makes it possible to notify in real time whether the subject is asleep or awake.

An example of a method of detecting dozing in the detection device 100 according to the first embodiment will be described with reference to the flowchart of FIG. 7 . The flowchart of FIG. 7 is executed by the processing unit 10 of the arithmetic unit 1. FIG. As an example, the detection method according to the first embodiment is started when R-wave data is obtained.

Referring to FIG. 7, processing unit 10 first initializes parameters T1, T2, f, and st used for processing (step S101). The parameters T1 and T2 are the duration time exceeding the control limit value and the duration time falling below the control limit value, respectively, for at least one of the Q statistic and the T2 statistic obtained as index values. The parameter f is a parameter representing whether the state is a state in which there is a possibility of falling asleep or a state in which there is a possibility of being awake, obtained from the index value and the first threshold. A parameter st is a parameter representing the determination result of whether the subject is asleep or awake.

Initial values of the parameters T1 and T2 are zero. The initial value of parameter f is 0, which indicates the possibility of arousal. The initial value of the parameter st is 0 representing the determination result of wakefulness.

The processing unit 10 acquires the RRI (step S103), and uses the RRI to calculate the HRV index and the Poincare index (step S105). The processing unit 10 performs canonical correlation analysis on a data set whose elements are the values of at least some of the 13 indices (step S107), and obtains a data set containing canonical variables.

The processing unit 10 dynamizes the data set obtained in step S107 (step S109) to generate a data set having time-series changes. After normalizing the obtained data set (step S111), the processing unit 10 inputs it to the model 105, which is a discriminator using MSPC (step S113). At least one of the Q statistic and the T2 statistic is obtained as an index value (step S115).

The processing unit 10 executes determination processing using the statistic used as the index value (step S300). By repeating the above process, the processing unit 10 determines in real time whether the subject is asleep or awake represented by the obtained R-wave data.

As for the determination process in step S300, first, the processing unit 10 compares the index value with the first threshold (step S301), referring to the flowchart of FIG. In the detection method according to the first embodiment, the statistic used as the index value is compared with the control limit value.

If the index value exceeds the first threshold (YES in step S301) and the parameter f representing the possibility of the state is 1, that is, if it indicates dozing off (YES in step S303), the processing unit 10 , the RRI at that time is added to the duration DT1 during which the index value exceeds the first threshold (step S305).

If the index value exceeds the first threshold (YES in step S301) and the parameter f representing the possibility of the state is 0, that is, if it indicates awakening (NO in step S303), the processing unit 10 , the duration DT1 is initialized, and the parameter f is changed to 1 indicating dozing off (step S307), and the RRI at that time is added to the duration DT1 (step S305). As a result, the measurement of the duration DT2 in a state in which there is a possibility of being awake is completed, and the measurement of the duration DT1 in a state in which there is a possibility of falling asleep is started.

The processing unit 10 compares the duration DT1 with a duration threshold t0, which is a second threshold. As a result, if the duration DT1 exceeds the threshold value t0 (YES in step S309), the processing unit 10 determines that the user is dozing, and sets the parameter st indicating the determination result to 1 indicating dozing (step S311).

On the other hand, if the index value is below the first threshold (NO in step S301) and the parameter f representing the possibility of the state is 0, that is, if it indicates awakening (YES in step S313), the processing unit 10 adds the RRI at that time to the duration DT2 during which the index value is below the first threshold (step S315).

If the index value is below the first threshold (NO in step S301) and the parameter f representing the possibility of the state is 1, that is, if it indicates dozing off (NO in step S313), the processing unit 10 , the duration DT2 is initialized, and the parameter f is changed to 0 indicating arousal (step S307), and the RRI at that time is added to the duration DT2 (step S315). As a result, the measurement of the duration DT1 in a state in which there is a possibility of dozing off is completed, and the measurement of the duration DT2 in a state in which there is a possibility of being awake is started.

The processing unit 10 compares the duration DT2 with a duration threshold t0, which is the second threshold. As a result, if the duration time DT2 exceeds the threshold value t0 (YES in step S319), the processing unit 10 determines that the user is awake, and sets the parameter st indicating the determination result to 0 indicating arousal (step S321).

If the parameter st indicating the determination result after the above processing is 1 indicating dozing off (YES in step S323), the processing unit 10 executes output processing such as outputting a warning (step S325).

By performing the determination process shown in FIG. 8 in the detection method according to the first embodiment, even when the statistic, which is the index value, exceeds the first threshold, the duration DT1 exceeds the second threshold. is not determined as falling asleep. Further, even if the index value falls below the first threshold after it is once determined as dozing off, it is not determined to have woken up unless the duration DT2 of the decrease does not exceed the second threshold. In other words, erroneous detection of dozing due to RRI noise or the like is prevented, and detection of wakefulness in a state of not being completely awake from a dozing state is also prevented. Therefore, dozing off of the subject is detected with high accuracy.

In the detection method according to the first embodiment, the inventor uses the T2 statistic and the Each of the Q-statistics was used to detect dozing. (A) of FIG. 9 represents the transition of the T2 statistic, and (B) represents the transition of the Q statistic for the same electrocardiographic signal. In FIGS. 9A and 9B, the horizontal axis represents time and the vertical axis represents index values. The control limit value T0 of the T2 statistic was set to 10, and the control limit value Q0 of the Q statistic was set to 0.2. In FIGS. 9A and 9B, the subject is dozing during period D from time T2 to time T3.

In the result of FIG. 9A, when the T2 statistic is used, the T2 statistic exceeds the control limit value T0 from time T0, and when the time T1 is reached, the duration DT becomes equal to or greater than the threshold t0, and dozing is detected. be done. On the other hand, as shown in FIG. 9B, in this measurement result, if the Q statistic is used, dozing is not detected even at time T1 when dozing is detected in (A).

Therefore, it was verified from the results of FIG. 9 that the T2 statistic is better when either the Q statistic or the T2 statistic is used as the index value.

Next, in the detection method according to the first embodiment, the inventor used only the Poincare index value when the HRV index value was directly used as the index value used as the data set to be input to the model 105. When using canonical variables obtained by canonical correlation analysis of the values of the HRV index and Poincare index, time-series data with time-series changes obtained by dynamizing the values of the HRV index and Poincare index When using time-series data with time-series changes obtained by dynamizing the canonical variables obtained by canonical correlation analysis of the values of the HRV index and Poincaré index, each detection accuracy is First to fifth tests were conducted for verification.

10 to 14 are diagrams showing ROC (Receiver Operating Characteristic) curves obtained by the first to fifth tests, respectively. The ROC curve is obtained by plotting TPR against FPR, with FPR (false positive rate) on the horizontal axis and TPR (true positive rate) on the vertical axis.

The inventors used the AUC (area under the receiver operating characteristic curve), sensitivity, and specificity obtained from the first to fifth test results to evaluate the data set input to the model 105, respectively. AUC is obtained by the area under the ROC curve. AUC takes a value from 0 to 1, and a larger value indicates higher detection performance. Sensitivity is the ratio of true positives to true positives and false negatives. Specificity is the rate of true negatives over false positives and true negatives.

In the first test, shown in Figure 10, an AUC of 0.8349, a sensitivity of 0.78 and a specificity of 0.71 was obtained. A second test, shown in FIG. 11, yielded an AUC of 0.8650, a sensitivity of 0.78 and a specificity of 0.88. In the third test, shown in Figure 12, an AUC of 0.8434, a sensitivity of 0.78 and a specificity of 0.71 was obtained. In the fourth test, shown in Figure 13, an AUC of 0.8534, a sensitivity of 0.89 and a specificity of 0.71 was obtained. In the fifth test, shown in Figure 14, an AUC of 0.8904, a sensitivity of 0.89 and a specificity of 0.75 was obtained.

When comparing the results of the first test and the second test, the second test has higher AUC and specificity than the first test, and the Poincare index is used to detect higher detection accuracy. Do you get it. Comparing the results of the 1st test and the 3rd to 5th tests, when using canonical variables obtained by canonical correlation analysis of the values of the HRV index and the Poincare index, When using time-series data with time-series changes, and when using time-series data with time-series changes obtained by dynamizing the canonical variables obtained by canonical correlation analysis, It was found that the detection accuracy is higher than when the HRV index value is used as it is.

Furthermore, when comparing the results of the 3rd to 5th tests, the time-series changes obtained by dynamizing the values of the HRV index and the Poincaré index using canonical variables obtained by canonical correlation analysis When using time-series data with time-series changes obtained by dynamizing It was found that the detection accuracy is higher when using time-series data with series changes.

From the above results, in the detection method according to the first embodiment, the detection accuracy is improved by using the value of the Poincare index as at least a part of the index value, rather than using the value of the HRV index as it is. Verified. Furthermore, it was verified that the use of canonical variables obtained by performing canonical correlation analysis on the index values increases the detection accuracy rather than using the index values as they are. In addition, it was verified that using time-series data with time-series changes obtained by dynamizing the index values improves the detection accuracy rather than using the index values as they are. Furthermore, rather than using the index values as they are, it is better to use time-series data with time-series changes obtained by dynamizing the canonical variables obtained by canonical correlation analysis of the index values. It was verified that the accuracy was improved.

In addition, in dynamization, in order to verify how many beats after the first data set the second data set to be linked to the first data set, the inventors The detection accuracy of the detection device 100 was evaluated by changing the difference in the number of beats of the data set from the first data set. AUC was used for evaluation of detection accuracy. As a result, the results shown in FIG. 15 were obtained. In FIG. 15, the vertical axis represents AUC and the horizontal axis represents the delay in beats of the second data set from the first data set. From the results of FIG. 15, it can be seen that in this test, the detection accuracy is highest when the second data set, which is eight beats behind the first data set, is connected to the first data set. From this, it was verified that it is preferable to use data with a delay of about 8 beats as the second data set.

[Second embodiment]

In the detection device 100 according to the second embodiment, the model 105 is a machine-learned model, such as an autoencoder. In the index value calculation process 102 of the second embodiment, the RRI is given as input data to the autoencoder, and the output data, which is the reconstructed data of the input data, is used to obtain the index value for doze detection. .

An autoencoder is an example of a deep learning model that uses LSTM (long short-term memory) and self-attention. It reconstructs input data and outputs values close to the input data. It is a deep learning model machine-learned using training data. Therefore, if the model 105 is an autoencoder, when data with a tendency different from the training data is input to the model 105, the error of the output data from the input data increases. That is, when abnormal data is input to the model 105 that has been machine-learned using normal data as training data, the error increases. The detection device 100 according to the second embodiment detects dozing by detecting abnormal data using this characteristic of the model 105 . This makes it possible to easily detect doze with high accuracy.

In the index value calculation process 102, the processing unit 10 uses a model 105, which is an autoencoder machine-learned using only the RRI data in wakefulness as training data. That is, the training data includes RRI data when awake and does not include RRI data when dozing. The model 105 may be stored in the computing device 1 or may be stored in another device accessible by the communication unit 13 .

The index value calculation process 102 according to the second embodiment includes inputting the RRI obtained from the R-wave data to the model 105 and obtaining output data. In the index value calculation process 102 , the processing unit 10 stores RRIs each time they are obtained, and inputs a predetermined number of RRIs to the model 105 . The predetermined number is 30, for example. At this time, 30 pieces of output data are obtained from the model 105 .

The index value calculation process 102 includes calculating the error of the output data from the model 105 from the RRI input to the model 105 . In the detection method according to the second embodiment, the obtained error is used as the index value. The error is, for example, the mean squared error.

The determination processing 103 according to the second embodiment includes comparing the error calculated by the index value calculation processing 102 with a first threshold and counting the duration DT1 exceeding the first threshold. The determination process 103 also includes timing the duration DT2 during which the error is below the first threshold. The first threshold value here is a value e0 representing the deviation from the input data to the extent that the data is considered to be abnormal data, and may be set in advance.

The processing unit 10 compares the error with the first threshold to determine whether the input RRI has the same tendency as the RRI obtained from the R-wave data during wakefulness or has a different tendency. be done. The threshold value e0, which is the first threshold value, is a value that can be used to determine that the input data is different from the input data obtained only when the user is awake, that is, to determine that the user is dozing off. Thus, when the error exceeds the threshold e0 and is large, it is determined that the subject is not awake, that is, the possibility of dozing off, and when the error is less than the threshold e0, the possibility that the subject is awake is determined. be.

Also in the determination processing 103 according to the second embodiment, the duration DT1 during which the error exceeds the threshold e0 and the duration DT2 during which the error is below the threshold e0 are measured. Then, determination processing 103 includes detecting the subject's dozing off when the obtained duration DT1 exceeds the second threshold, and detecting the subject's wakefulness when the obtained duration DT2 exceeds the second threshold. The second threshold is a duration time threshold t0 that can be determined as dozing or wakefulness, and is, for example, about 10 seconds.

As a result, the detection apparatus 100 according to the second embodiment also detects the subject's dozing off when the state in which the error exceeds the threshold value e0 continues for the duration time equal to or longer than the threshold value t0 that can be determined as dozing. Also, when the state in which the error is below the threshold value e0 continues for a duration equal to or longer than the threshold value t0, which can be determined as awakening, the awakening of the subject is detected. As a result, dozing can be detected with higher accuracy than judging dozing when the error exceeds the first threshold or judging awakening when the error falls below the first threshold.

An example of a method of detecting dozing in the detection device 100 according to the second embodiment will be described with reference to the flowchart of FIG. 16 . The flowchart of FIG. 16 is executed by the processing unit 10 of the arithmetic unit 1. FIG. The detection method according to the second embodiment also starts, as an example, when R-wave data is obtained.

Referring to FIG. 16, processing unit 10 first initializes parameters T1, T2, f, and st used in processing (step S201).

The processing unit 10 acquires the RRI (step S203), standardizes it, and stores it in the memory (step S205). The processing unit 10 repeats the above calculation of RRI until the RRI stored in the memory reaches a number suitable for input to the model 105, which is an autoencoder (NO in step S207).

When the number of RRIs stored in memory reaches a number suitable for input to model 105 (YES in step S207), processing unit 10 inputs these RRIs to model 105, which is an autoencoder (step S209). As a result, values obtained by reconstructing the RRI from the model 105 are obtained as output data. The processing unit 10 calculates the error from the output data of the input RRI model 105 (step S211) and uses it as an index value.

The processing unit 10 executes determination processing using the error as the index value (step S300). The determination process of step S300 is the process of FIG. By repeating the above process, the processing unit 10 determines in real time whether the subject is asleep or awake represented by the obtained R-wave data. Here, by using the RRI as an input to the model 105 without calculating the HRV index etc. from the RRI, the index value can be easily obtained, and the amount of information can be reduced when calculating various indices from the RRI. It can be prevented and the detection accuracy can be improved.

Further, even when the error, which is the index value, exceeds the first threshold value, the duration time DT1 is reduced to the second threshold value by performing the determination process shown in FIG. If it does not exceed the threshold, it is not judged as dozing. Further, even if the index value falls below the first threshold after it is once determined as dozing off, it is not determined to have woken up unless the duration DT2 of the decrease does not exceed the second threshold. In other words, erroneous detection of dozing due to RRI noise or the like is prevented, and detection of wakefulness in a state of not being completely awake from a dozing state is also prevented. Therefore, dozing off of the subject is detected with high accuracy.

[Evaluation test]

The inventor conducted an evaluation test for evaluating the detection accuracy of the detection method according to the second embodiment. In the evaluation test, 0.6 was used as the error threshold value e0, which is an index value, and 10 seconds was used as the duration time threshold value t0.

17 and 18 are diagrams showing the results of evaluation experiments on the detection method according to the second embodiment. The axis represents time. FIG. 17 shows detection results from electrocardiogram signals including the state in which the subject is asleep, and FIG. 18 shows detection results from electrocardiogram signals in which the subject is always awake. In these figures, the gray area indicates the period during which the subject's dozing was detected by the detection device 100 .

Specifically, in the error transition shown in FIG. 17, there is a range in which the error exceeds the threshold value e0 for 10 seconds or more, and dozing is detected in that range. After that, when the period during which the error is below the threshold value e0 exceeds 10 seconds, awakening from dozing is detected. On the other hand, in the transition of the error shown in FIG. 18, although there was a timing when the error exceeded the threshold value e0, the state of exceeding did not continue for 10 seconds or more, so dozing was not detected.

From the results of FIGS. 17 and 18, even if the detection method according to the second embodiment causes the error to exceed the threshold value e0 representing the state of dozing due to noise, fluctuations in the heartbeat, etc., It prevents you from being detected. That is, it is possible to improve the accuracy of detecting dozing.

The inventors performed an evaluation test of the detection method according to the second embodiment for each of the LSTM and self-attention autoencoder configurations, and evaluated the autoencoder configurations. In this evaluation test, 26 subjects performed a driving experiment using a driving simulator, and the RRI calculated from the electrocardiographic signals obtained during the experiment was used as a data set. Each datum has a length of 15-20 minutes.

The AUC (area under the receiver operating characteristic curve) of the dataset used was used to evaluate the configuration of the autoencoder. AUC is a positive predictive value (TPS) and a false positive rate (FPS) corresponding to each threshold e0 when the threshold t0 of the duration is fixed and the threshold e0 of the error is changed. ) is obtained by the area under the trajectory drawn by The trajectory drawn by the positive predictive value and the false positive rate is called the ROC curve. AUC takes a value from 0 to 1, and a larger value indicates higher detection performance. In the evaluation test, the threshold t0 was varied to obtain the AUC at each value of the threshold t0.

FIG. 19 is a graph showing the results of evaluation tests for LSTM and self-attention autoencoder configurations, with the vertical axis representing the AUC and the horizontal axis representing the value of the threshold value t0. From the results of FIG. 19, it was verified that the self-attention configuration of the autoencoder provided higher detection accuracy than the LSTM configuration, regardless of the value of the threshold value t0.

<3. Note>
The present invention is not limited to the above embodiments, and various modifications are possible.

1: Arithmetic device 2: Heart rate measuring device 10: Processing unit 12: Memory 13: Communication unit 21: Electrodes 100: Detection device 101: RRI calculation processing 102: Index value calculation processing 103: Judgment processing 104: Output processing 105: Model 121 : program A1 : major axis A2 : minor axis D : period D1 : first dataset D2 : second dataset D3 : third dataset E : ellipse HF : correction I : signal intensity Ith : intensity threshold LF : Correction P: Subject Q0: Control limit value T: Duration T0: Control limit value e0: Threshold value t0: Threshold value

Claims (12)

A detection device for detecting dozing,
Equipped with a computing device that executes detection processing for detecting dozing,
The detection process includes
Obtaining an index value for doze detection from the RRI (RR Interval) generated from the electrocardiographic signal of the subject,
timing the duration that the index value exceeds a first threshold;
detecting doze of the subject when the duration exceeds a second threshold. Obtaining the index value for the doze detection involves providing a value obtained from the RRI as an input value to a model that uses the target data as an input value and outputs a value used for detecting an abnormality in the target data, 2. A doze detection device according to claim 1, comprising using an output value from the model to obtain the index value. The model is an autoencoder,
Obtaining the index value for the doze detection includes:
giving an RRI generated from the electrocardiographic signal of the subject as input data to an autoencoder to acquire output data that is reconstructed data of the input data;
The dozing detection device according to claim 2, further comprising: calculating an error of the output data from the RRI generated from the electrocardiographic signal of the subject as the index value. 4. The dozing detection device according to claim 3, wherein the autoencoder is learned by giving RRI generated from an electrocardiographic signal during wakefulness as input data for learning. The model is a discriminator using MSPC (Multivariate Statistical Process Control),
Obtaining the index value for the doze detection includes:
generating a heartbeat index from the RRI generated from the electrocardiographic signal of the subject;
The dozing detection device according to claim 2, comprising: inputting a data set including a plurality of variables HRV among the indices related to the heart rate to the discriminator, and obtaining a statistic as the index value for the doze detection. . 6. The dozing detection device according to claim 5, wherein the plurality of variables HRV include at least one of variables included in a Poincare index obtained from RRI generated from the subject's electrocardiogram signal. The dozing detection device according to claim 5 or 6, wherein the data set includes a plurality of canonical variables among the canonical variables obtained by canonical correlation analysis of the plurality of variables HRV. 8. The doze according to any one of claims 5 to 7, wherein the data set includes data having a time-series change generated from the index related to the heartbeat obtained from each of a plurality of heartbeats of the electrocardiographic signal of the subject. detection device. The data set is obtained from a plurality of variables HRV of the indices related to the heart rate, including at least one of variables included in a Poincaré index obtained from an RRI generated from the electrocardiographic signal of the subject. containing data with time series variation,
The data having the time-series change is obtained from each of the plurality of heartbeats of the electrocardiographic signal of the subject for each of the plurality of canonical variables among the canonical variables obtained by canonically analyzing the plurality of variables HRV. 6. The doze detection device according to claim 5, wherein the index is generated from the obtained index related to the heart rate. Obtaining the index value for the doze detection includes an autoencoder, a discriminator using MSPC, a deep learning model using long short-term memory (LSTM), and self-attention. A doze detection device according to claim 1, comprising using at least one of: A method for detecting drowsiness, comprising:
Obtaining an index value for doze detection from the RRI generated from the electrocardiogram signal of the subject,
timing the duration that the index value exceeds a first threshold;
detecting the subject falling asleep when the duration exceeds a second threshold. A computer program for causing a computer to execute a detection process for detecting dozing,
The detection process includes
Obtaining an index value for doze detection from the RRI generated from the electrocardiogram signal of the subject,
timing the duration that the index value exceeds a first threshold;
A computer program comprising detecting dozing off of the subject when the duration exceeds a second threshold.

Images