JP6879388B2 - Alertness estimation device, alertness estimation method, and program - Google Patents

Description

The present invention, for estimating the wakefulness representing the arousal state of a person, awakening level estimation apparatus, and relates to awakening level estimation method further relates to a program for realizing these.

In recent years, the working-age population has decreased due to the declining birthrate and aging population, and the labor shortage is progressing. Under such circumstances, attempts to replace some of the work that humans have done so far with robots or AI (artificial intelligence) are increasing. However, among the jobs performed by humans, it is difficult to replace them with robots or AI for jobs that require intellectual labor. For this reason, it will be essential for humans to maintain and improve the productivity of intellectual labor in the future.

By the way, unlike machines, humans may feel drowsy (low arousal state) or stress (hyperawakening state). In other words, the productivity of human intellectual labor changes according to the state of mental and physical arousal. Therefore, in order to improve the productivity of human intellectual labor, it is important to make the wakefulness just right.

Then, as a method for improving the wakefulness of a person during intellectual work, the latest wakefulness of a person is detected, and the temperature, humidity, illuminance, etc. in the office are adjusted according to the detected wakefulness. Controlling the environment can be mentioned. In particular, in this method, it is important to accurately detect the awake state of a person.

For example, Patent Document 1 discloses a device that estimates a person's alertness from the opening degree of an eye. The device disclosed in Patent Document 1 acquires the eye opening time of the driver's eyes from a camera image sent at a set frame rate, obtains the variation from the acquired eye opening time, and obtains the variation from the obtained variation. Estimate the degree of arousal.

In addition, Non-Patent Document 1 discloses an apparatus for estimating a person's alertness (stress) from a facial image. The device disclosed in Non-Patent Document 1 calculates a low-frequency HRV (Heart Rate Variability) component and a respiratory rate of a human face from a camera image sent at a set frame rate, and statistically calculates the calculated values. Input to the model to estimate a person's arousal level.

International Publication No. 2010/092860

Daniel McDuff, Sarah Gontarek, and Rosalind Picard, "RemoteMeasurement of Cognitive Stress via Heart Rate Variability", EMBC2014

By the way, in both the device disclosed in Patent Document 1 and the device disclosed in Non-Patent Document 1, it is necessary to set a high frame rate of the camera image and process it in order to maintain the estimation accuracy of the arousal degree. There is. Therefore, there is a problem that a large processing load is applied to the device.

An example of an object of the present invention is to provide an arousal level estimation device, an arousal level estimation method, and a program capable of accurately estimating a person's arousal level while solving the above problems and reducing the processing load. ..

In order to achieve the above object, the arousal level estimation device in one aspect of the present invention is a device for estimating the arousal level of the user.
An image data acquisition unit that acquires image data including the user's face image at a set frame rate, and
A time-series data extraction unit that extracts time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and a time-series data extraction unit.
A data processing unit that interpolates the time-series data so that the number of samples of the extracted time-series data becomes a set value.
An arousal level estimation unit that estimates the arousal level of the user by inputting the interpolated time series data into a learning model constructed using a convolutional neural network.
It is characterized by having.
It is characterized by that.

Further, in order to achieve the above object, the arousal level estimation method in one aspect of the present invention is a method for estimating the arousal level of the user.
(A) A step of acquiring image data including the user's face image at a set frame rate, and
(B) A step of extracting time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and
(C) A step of interpolating the time-series data so that the number of samples of the extracted time-series data becomes a set value.
(D) A step of inputting the interpolated time series data into a learning model constructed by using a convolutional neural network to estimate the arousal degree of the user.
It is characterized by having.

Furthermore, in order to achieve the above object, a program according to an aspect of the present invention is a program for estimating the arousal level of the user by a computer,
On the computer
(A) A step of acquiring image data including the user's face image at a set frame rate, and
(B) A step of extracting time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and
(C) A step of interpolating the time-series data so that the number of samples of the extracted time-series data becomes a set value.
(D) A step of inputting the interpolated time series data into a learning model constructed by using a convolutional neural network to estimate the arousal degree of the user.
Ru is the execution, program.

As described above, according to the present invention, it is possible to accurately estimate the arousal level of a person while reducing the processing load.

FIG. 1 is a block diagram showing a configuration of an alertness estimation device according to the first embodiment of the present invention. FIG. 2 is a diagram showing an example of interpolation of time series data performed in the first embodiment of the present invention. FIG. 3 is a diagram showing an example of a convolutional neural network used in the first embodiment of the present invention. FIG. 4 is a flow chart showing the operation of the alertness estimation device 10 according to the first embodiment of the present invention. FIG. 5 is a block diagram showing the configuration of the alertness estimation device according to the second embodiment of the present invention. FIG. 6 is a flow chart showing the operation of the alertness estimation device 30 according to the second embodiment of the present invention. FIG. 7 is a block diagram showing an example of a computer that realizes the alertness estimation device according to the first and second embodiments of the present invention.

(Embodiment 1)
Hereinafter, the arousal level estimation device, the arousal level estimation method, and the program according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 4.

[Device configuration]
First, the configuration of the alertness estimation device according to the first embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of an alertness estimation device according to the first embodiment of the present invention.

The arousal level estimation device 10 according to the present embodiment shown in FIG. 1 is a device for estimating the arousal level of the user. As shown in FIG. 1, the arousal level estimation device 10 includes an image data acquisition unit 11, a time series data extraction unit 12, a data processing unit 13, and an arousal level estimation unit 14.

Of these, the image data acquisition unit 11 acquires image data including the user's face image at the set frame rate. In addition, the time-series data extraction unit 12 extracts time-series data indicating the user's biological information from the image data acquired at the set frame rate.

The data processing unit 13 interpolates the time-series data so that the number of samples of the extracted time-series data becomes a set value. The arousal level estimation unit 14 inputs the time-series data after interpolation into the learning model constructed by using the convolutional neural network, and estimates the arousal level of the user.

As described above, in the first embodiment, since the sampling number of the time series data extracted from the image data can be interpolated, the frame rate of the image data can be suppressed low in advance. Therefore, according to the first embodiment, it is possible to accurately estimate the arousal level of a person while reducing the processing load on the apparatus.

Subsequently, the configuration of the alertness estimation device 10 according to the first embodiment will be described more specifically. First, as shown in FIG. 1, in the first embodiment, the alertness estimation device 10 is connected to an external imaging device 20.

The image pickup device 20 is a digital camera and outputs image data at a set frame rate. Further, the image pickup device 20 is arranged so that the user's face image can be taken. The image data acquisition unit 11 acquires image data output from the image pickup apparatus 20.

Further, in the first embodiment, the biological information indicated by the time series data includes information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, information indicating the direction of the face, and information indicating the pulse wave in the user. Information indicating blood flow, information indicating the degree of opening and closing of the mouth, and the like can be mentioned. In the first embodiment, the time-series data extraction unit 12 extracts the above-mentioned information from the image data for each frame and generates time-series data.

Specifically, the time-series data extraction unit 12 detects, for example, the user's eyes from the image data, obtains the degree of opening / closing from the detected size of both eyes, and time-series of information indicating the degree of opening / closing of the eyes. Generate data. Further, the time-series data extraction unit 12 detects the center positions of both eyes of the user from the image data, calculates the direction of the line of sight from each of the detected center positions, and time-series data of information indicating the calculated direction of the line of sight. To generate.

Further, the time-series data extraction unit 12 detects the center line and contour of the user's face from the image data, calculates the orientation of the user's face from the positional relationship of the detected center line and contour, and calculates the face orientation of the calculated face. Generate time-series data of information indicating the direction. Further, the time-series data extraction unit 12 detects the user's mouth from the image data, obtains the opening / closing degree from the detected mouth size, and generates time-series data of information indicating the opening / closing degree of the mouth.

In addition, the time-series data extraction unit 12 calculates the pulse wave or blood flow of the user by using the property that hemoglobin in the blood absorbs the green component of light (see, for example, the following references). Specifically, the time-series data extraction unit 12 identifies the area of the user's skin from the image data, and calculates the brightness values of the R, G, and B channels in the specified area. Then, the time-series data extraction unit 12 calculates the pulse wave or blood flow of the user by utilizing the fact that green light is absorbed when the blood flow increases and the brightness value of G decreases, and the pulse wave or blood is calculated. Generate time-series data of information showing blood flow.

References: Asami Umematsu, Takenori Tsujikawa, "High-precision heart rate estimation method from facial images based on ICA-R", NEC Data Science Laboratory, Institute of Electronics, Information and Communication Engineers, IEICE Technical Report 2017

Further, in the first embodiment, the data processing unit 13 interpolates the time-series data by upsampling the time-series data as shown in FIG. FIG. 2 is a diagram showing an example of interpolation of time series data performed in the first embodiment of the present invention. In the example of FIG. 2, the originally required frame rate of the time series data is R, and the actual frame rate of the time series data is (R / 2). In addition, "○" in the figure indicates time series data.

As shown in FIG. 2, the data processing unit 13 receives two consecutive time-series data so that the number of samplings becomes a set value, that is, the frame rate changes from (R / 2) to R. Add new time series data in between.

Further, new time series data is added by, for example, linear interpolation or spline interpolation. Further, for example, a neural network may be constructed by using data having a frame rate of R / 2 and data having a frame rate of R as learning data, and interpolation may be performed by this neural network. Then, as shown in FIG. 2, the upsampled time series data is input to the convolutional neural network by the alertness estimation unit 14.

In addition, the data processing unit 13 can also perform various signal processing, for example, noise removal processing, missing data interpolation processing, outlier removal processing, and the like. By such processing, it can be expected that the arousal degree estimation unit 14 will improve the estimation accuracy of the arousal degree later.

Further, in the first embodiment, when the biological information indicated by the time series data is two or more pieces of information, the learning model has a layer for convolving each biological information. Here, the estimation process by the alertness estimation unit 14 in the first embodiment will be specifically described with reference to FIG.

FIG. 3 is a diagram showing an example of a convolutional neural network used in the first embodiment of the present invention. In the example of FIG. 3, the time-series data shows three pieces of information such as information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, and information indicating the direction of the face, and convolution is performed for each information. .. Further, it is assumed that the time series data is normalized to a value such as 0 to 1 or -1 to +1 for each time series.

As shown in FIG. 3, the size of the time-series data showing the opening and closing degree of eyes and (D _{EC, W T). DEC} is the number of time-series data indicating the degree of opening and closing of the eye. Further, for example, to enter the time-series data for each of the opening and closing right and left eyes, a D _{EC = 2.} W _T is the time window width for the awakening level estimation. For example, when using the 10s of data, by using the frame rate _R, a W T = 10R.

First, the awakening level estimation unit 14, the size _(D EC, _{W T)} when type series data to convolution layer, size _(D EC, _{W T)} for each smaller windows, convolution weight filter, a bias In addition, the output is obtained through the activation function. Next, the alertness estimation unit 14 shifts the position of the window and performs the same operation using different weight filters and biases to obtain an output of _{size (D EC_C1} , W _{EC_C1).} Examples of the activation function include ReLU (Rectified Linear Unit).

Next, the awakening level estimation unit 14, in pooling layer, the size _{(D EC_C1, W EC_C1)} for the input data, size _{(D EC_C1, W EC_C1)} for each smaller windows, the pooling process in pooling layer Do. For example, in the case of max pooling, which is often used, processing is performed so that the maximum value in the window remains. Next, the alertness estimation unit 14 shifts the position of the window and performs the same pooling process to obtain an output _{of the size (D EC_P1} , W _{EC_P1).}

Subsequently, the arousal level estimating unit 14, similarly in the next convolution layer, the size _{(D EC_P1, W EC_P1)} for the input data, size _{(D EC_P1, W EC_P1)} for each smaller windows, the filter Convolution, addition of bias, input to activation function. Then, in this case as well, the alertness estimation unit 14 shifts the position of the window and performs the same processing to obtain an output _{of the size (D EC_C2} , W _{EC_C2).}

Furthermore, the awakening level estimation unit 14, similarly in the following pooling layer, the size _{(D EC_C2, W EC_C2)} for the input data, size _{(D EC_C2, W EC_C2)} for each smaller windows, the pooling process Do. Next, the alertness estimation unit 14 shifts the position of the window and performs the same pooling process to obtain an output _{of, for example, a size (DEC_P2, 1).}

Further, in the first embodiment, the alertness estimation unit 14 performs the same processing on the time-series data indicating the direction of the line of sight and the time-series data indicating the direction of the face, and for example, the size (D). _{The output of EG_P2} , 1) and the output of size ( _{DHP_P2} , 1) are obtained.

Next, the alertness estimation unit 14 concatenates and flattens the outputs of each time-series data as a “concatenation & flattening” process (see FIG. 3). As a result, the alertness estimation unit 14 obtains an output of _{the size (1, D EC_P2} + D _{EG_P2} + D _{HP_P2).}

After that, the alertness estimation unit 14 convolves the weight filter, biases it, and passes it through the activation function for all the input data _{of the size (1, D EC_P2} + D _{EG_P2} + D _{HP_P2) in the fully connected layer.} , Get the alertness estimate as an output.

Further, in the first embodiment, the learning model uses a convolutional neural network, but the weight filter and the bias in the convolutional layer use the sample data to which the correct answer label of the arousal degree is given in advance. Learned by doing deep learning. Further, the learning can be performed by using an error back propagation method or the like so that the difference between the correct label of the arousal degree and the estimated value of the arousal degree becomes small.

Furthermore, in learning, overfitting can be prevented by using Dropout, which learns with a certain probability that the weight and bias are set to zero. Further, in the first embodiment, the learning model is a plurality of time series data having different frame rates (for example, time series data having frame rates of R, R / 2, R / 3, R / 6, and R / 10). ), The data obtained by interpolating so that the number of samples becomes a set value may be constructed by inputting it into a convolutional neural network as training data. In this case, more precise modeling becomes possible.

Further, in the learning model used in the first embodiment, the configuration of the convolutional neural network is not particularly limited. The learning model may have, for example, a configuration in which the second pooling layer is removed and instead a further fully connected layer is added after the connection & flattening. Various modifications may be added to the learning model.

[Device operation]
Next, the operation of the alertness estimation device 10 in the first embodiment will be described with reference to FIG. FIG. 4 is a flow chart showing the operation of the alertness estimation device 10 according to the first embodiment of the present invention. In the following description, FIGS. 1 to 3 will be referred to as appropriate. Further, in the first embodiment, the alertness estimation method is implemented by operating the alertness estimation device 10. Therefore, the description of the arousal level estimation method in the first embodiment is replaced with the following operation description of the arousal level estimation device 10.

As shown in FIG. 4, when the image data is output from the image pickup apparatus 20, the image data acquisition unit 11 acquires the output image data and holds the acquired image data (step S1).

Next, the image data acquisition unit 11 determines whether or not the number of held image data has reached a predetermined value (step S2). As a result of the determination in step S2, if the number of image data has not reached the predetermined value, the image data acquisition unit 11 executes step S1 again. On the other hand, as a result of the determination in step S2, when the number of image data reaches a predetermined value, the image data acquisition unit 11 passes the held image data to the time series data extraction unit 12.

Next, when the time-series data extraction unit 12 receives the image data, the time-series data extraction unit 12 extracts the time-series data indicating the biometric information of the user from the image data acquired in step S1 (step S3). When the image data includes a plurality of users, the time-series data extraction unit 12 can also extract the time-series data for each user in step S3.

Next, the data processing unit 13 interpolates the time-series data so that the number of samples of the time-series data extracted in step S3 becomes the set value (step S4).

Next, the arousal level estimation unit 14 inputs the time-series data after being interpolated by step S4 into the learning model constructed using the convolutional neural network, and estimates the user's arousal level (step S5). ..

Specifically, as shown in FIG. 3, when the time-series data shows information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, and information indicating the direction of the face, the alertness estimation unit 14 , For each piece of information, convolution is performed to estimate the arousal level. Further, after the execution of step S5, steps S1 to S5 are executed again, and the user's arousal level is constantly estimated.

Further, the arousal level estimation device 10 inputs the estimated arousal level to the control system of the air conditioner, the operation system of the vehicle, and the like. As a result, each system can perform optimization control based on the arousal level of the user.

[Effect in Embodiment 1]
As described above, in the first embodiment, since the sampling number is interpolated in the time series data extracted from the image instead of the image data, the frame rate of the image data can be suppressed low in advance. Therefore, the processing of the time-series data extraction unit 12 from the image data, which is a heavy burden, can be reduced, and as a result, the arousal level of a person can be estimated accurately while reducing the processing load in the entire device. Further, in the first embodiment, the estimation accuracy of the arousal degree can be further improved by modeling the convolutional neural network using the data interpolated for the input time series data of a plurality of frame rates. .. Further, in the first embodiment, since a plurality of biological information can be used as the time series data, the accuracy of the arousal degree can be further improved.

[program]
The program according to the first embodiment may be any program that causes a computer to execute steps S1 to S5 shown in FIG. By installing this program on a computer and executing it, the arousal degree estimation device 10 and the arousalness degree estimation method according to the first embodiment can be realized. In this case, the computer processor functions as an image data acquisition unit 11, a time series data extraction unit 12, a data processing unit 13, and an arousal degree estimation unit 14 to perform processing.

Further, the program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the image data acquisition unit 11, the time series data extraction unit 12, the data processing unit 13, and the arousal degree estimation unit 14.

(Embodiment 2)
Next, the alertness estimation device according to the second embodiment of the present invention will be described with reference to FIGS. 5 and 6.

[Device configuration]
First, the configuration of the alertness estimation device according to the second embodiment will be described with reference to FIG. FIG. 5 is a block diagram showing the configuration of the alertness estimation device according to the second embodiment of the present invention.

As shown in FIG. 5, the alertness estimation device 30 according to the second embodiment includes a frame rate adjusting unit 31 in addition to the same configuration as the alertness estimation device 10 according to the first embodiment shown in FIG. ing. Hereinafter, the differences from the first embodiment will be mainly described.

The frame rate adjusting unit 31 adjusts the frame rate according to the arousal level estimated by the arousal level estimation unit 14. Further, the frame rate adjusting unit 31 instructs the image pickup apparatus 20 that outputs the image data after adjusting the frame rate to the adjusted frame rate. Further, the frame rate adjusting unit 31 may instruct the image data acquisition unit 11 and the time series data extraction unit 12 instead of the image pickup apparatus 20 to indicate the adjusted frame rate. Further, the frame rate adjusting unit 31 can also adjust the frame rate according to the biometric information indicated by the extracted time series data.

Specifically, when the arousal level is constant, the frame rate adjusting unit 31 sets the frame rate low to reduce the processing load on the arousal level estimation device 30. On the other hand, when the arousal degree has changed significantly (when the range of change exceeds a predetermined range), the frame rate adjusting unit 31 sets the frame rate high to improve the estimation accuracy of the arousal degree.

[Device operation]
Next, the operation of the alertness estimation device 30 in the second embodiment will be described with reference to FIG. FIG. 6 is a flow chart showing the operation of the alertness estimation device 30 according to the second embodiment of the present invention. In the following description, FIG. 5 will be referred to as appropriate. Further, in the second embodiment, the alertness estimation method is implemented by operating the alertness estimation device 30. Therefore, the description of the arousal level estimation method in the second embodiment is replaced with the following operation description of the arousal level estimation device 30.

As shown in FIG. 6, when the image data is output from the image pickup apparatus 20, the image data acquisition unit 11 acquires the output image data and holds the acquired image data (step S11).

Next, the image data acquisition unit 11 determines whether or not the number of held image data has reached a predetermined value (step S12). As a result of the determination in step S12, if the number of image data has not reached the predetermined value, the image data acquisition unit 11 executes step S11 again. On the other hand, as a result of the determination in step Ss2, when the number of image data reaches a predetermined value, the image data acquisition unit 11 passes the held image data to the time series data extraction unit 12.

Next, when the time-series data extraction unit 12 receives the image data, the time-series data extraction unit 12 extracts the time-series data indicating the biometric information of the user from the image data acquired in step S11 (step S13). When the image data includes a plurality of users, the time-series data extraction unit 12 can also extract the time-series data for each user in step S13.

Next, the data processing unit 13 interpolates the time-series data so that the number of samples of the time-series data extracted in step S13 becomes the set value (step S14).

Next, the arousal level estimation unit 14 inputs the time-series data after being interpolated by step S14 into the learning model constructed using the convolutional neural network, and estimates the user's arousal level (step S15). ..

By executing the above steps S11 to S15, the user's arousal level is estimated. Steps S11 to S15 are the same steps as steps S1 to S5 shown in FIG.

Next, after the execution of step S15, the frame rate adjusting unit 31 adjusts the frame rate according to the arousal level estimated by step S15 (step S16). Subsequently, the frame rate adjusting unit 31 instructs the image pickup apparatus 20 of the frame rate adjusted in step S16 (step S17).

After the execution of step S17, the image pickup apparatus 20 outputs the image data at the instructed frame rate. Further, after the execution of step S17, steps S11 to S17 are executed again, and at that time, time series data is generated at the instructed frame rate, and the arousal level is newly estimated. Further, also in the second embodiment, the user's arousal level is always estimated by executing steps S11 to S15 again.

Further, also in the second embodiment, the arousal level estimation device 30 inputs the estimated arousal level to the control system of the air conditioner, the vehicle operation system, and the like. As a result, each system can perform optimization control based on the arousal level of the user.

[Effect in Embodiment 2]
As described above, in the second embodiment, the frame rate of the image data can be adjusted. According to the second embodiment, an appropriate frame rate can be set according to the required accuracy of the arousal degree. Further, also in the second embodiment, the same effect as that of the first embodiment can be obtained.

[program]
The program according to the second embodiment may be any program that causes the computer to execute steps S11 to S17 shown in FIG. By installing this program on a computer and executing it, the arousal degree estimation device 30 and the arousalness degree estimation method according to the first embodiment can be realized. In this case, the computer processor functions as an image data acquisition unit 11, a time series data extraction unit 12, a data processing unit 13, an arousal degree estimation unit 14, and a frame rate adjustment unit 31 to perform processing.

Further, the program in the second embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the image data acquisition unit 11, the time series data extraction unit 12, the data processing unit 13, the arousal degree estimation unit 14, and the frame rate adjustment unit 31, respectively. good.

(Physical configuration)
Here, a computer that realizes an alertness estimation device by executing the programs according to the first and second embodiments of the present invention will be described with reference to FIG. 7. FIG. 7 is a block diagram showing an example of a computer that realizes the alertness estimation device according to the first and second embodiments of the present invention.

As shown in FIG. 7, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. And. Each of these parts is connected to each other via a bus 121 so as to be capable of data communication. The computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-ProgrammableGate Array) in addition to the CPU 111 or in place of the CPU 111.

The CPU 111 expands the programs (codes) of the present embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various operations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Further, the program according to the present embodiment is provided in a state of being stored in a computer-readable recording medium 120. The program in the present embodiment may be distributed on the Internet connected via the communication interface 117.

Further, specific examples of the storage device 113 include a semiconductor storage device such as a flash memory in addition to a hard disk drive. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader / writer 116 mediates the data transmission between the CPU 111 and the recording medium 120, reads the program from the recording medium 120, and writes the processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

Specific examples of the recording medium 120 include a general-purpose semiconductor storage device such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), a magnetic recording medium such as a flexible disk, or a CD-. Examples include optical recording media such as ROM (Compact DiskRead Only Memory).

The arousal level estimation device in the present embodiment can also be realized by using the hardware corresponding to each part instead of the computer in which the program is installed. Further, the alertness estimation device may be partially realized by a program and the rest may be realized by hardware.

A part or all of the above-described embodiments can be expressed by the following descriptions (Appendix 1) to (Appendix 18), but the present invention is not limited to the following description.

(Appendix 1)
A device for estimating the user's alertness,
An image data acquisition unit that acquires image data including the user's face image at a set frame rate, and
A time-series data extraction unit that extracts time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and a time-series data extraction unit.
A data processing unit that interpolates the time-series data so that the number of samples of the extracted time-series data becomes a set value.
An arousal level estimation unit that estimates the arousal level of the user by inputting the interpolated time series data into a learning model constructed using a convolutional neural network.
An arousal level estimation device characterized by being equipped with.

(Appendix 2)
The arousal level estimation device according to Appendix 1.
The training model is constructed by inputting data obtained by interpolating a plurality of time series data having different frame rates so that the number of samples becomes a set value into a convolutional neural network as training data. Has been
An arousal level estimation device characterized by this.

(Appendix 3)
The arousal level estimation device according to Appendix 1 or 2.
The biological information indicated by the time-series data includes information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, information indicating the direction of the face, information indicating pulse waves, information indicating blood flow, and information of the mouth of the user. At least one of the information indicating the degree of opening / closing,
An arousal level estimation device characterized by this.

(Appendix 4)
The arousal level estimation device described in Appendix 3,
When the biometric information indicated by the time series data is two or more pieces of information, the learning model has a layer for convolution for each of the biometric information.
An arousal level estimation device characterized by this.

(Appendix 5)
The arousal level estimation device according to any one of Appendix 1 to 4.
It further includes a frame rate adjusting unit that adjusts the frame rate according to the estimated alertness.
An arousal level estimation device characterized by this.

(Appendix 6)
The arousal level estimation device according to Appendix 5.
The frame rate adjusting unit further adjusts the frame rate according to the biometric information indicated by the extracted time series data.
An arousal level estimation device characterized by this.

(Appendix 7)
A method for estimating user alertness,
(A) A step of acquiring image data including the user's face image at a set frame rate, and
(B) A step of extracting time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and
(C) A step of interpolating the time-series data so that the number of samples of the extracted time-series data becomes a set value.
(D) A step of inputting the interpolated time series data into a learning model constructed by using a convolutional neural network to estimate the arousal degree of the user.
A method for estimating arousal level, which comprises.

(Appendix 8)
The arousal level estimation method described in Appendix 7
The training model is constructed by inputting data obtained by interpolating a plurality of time series data having different frame rates so that the number of samples becomes a set value into a convolutional neural network as training data. Has been
An alertness estimation method characterized by this.

(Appendix 9)
The arousal level estimation method described in Appendix 7 or 8, wherein the arousal level is estimated.
The biological information indicated by the time-series data includes information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, information indicating the direction of the face, information indicating pulse waves, information indicating blood flow, and information of the mouth of the user. At least one of the information indicating the degree of opening / closing,
An alertness estimation method characterized by this.

(Appendix 10)
The arousal level estimation method described in Appendix 9,
When the biometric information indicated by the time series data is two or more pieces of information, the learning model has a layer for convolution for each of the biometric information.
An alertness estimation method characterized by this.

(Appendix 11)
The arousal level estimation method according to any one of Appendix 7 to 10.
(E) Further having a step of adjusting the frame rate according to the estimated alertness.
An alertness estimation method characterized by this.

(Appendix 12)
The arousal level estimation method described in Appendix 11,
In the step (e), the frame rate is further adjusted according to the biometric information indicated by the extracted time series data.
An alertness estimation method characterized by this.

(Appendix 13)
A program for estimating the arousal level of the user by a computer,
On the computer
(A) A step of acquiring image data including the user's face image at a set frame rate, and
(B) A step of extracting time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and
(C) A step of interpolating the time-series data so that the number of samples of the extracted time-series data becomes a set value.
(D) A step of inputting the interpolated time series data into a learning model constructed by using a convolutional neural network to estimate the arousal degree of the user.
Ru is the execution, program.

(Appendix 14)
The program described in Appendix 13
The training model is constructed by inputting data obtained by interpolating a plurality of time series data having different frame rates so that the number of samples becomes a set value into a convolutional neural network as training data. Has been
A program characterized by that.

(Appendix 15)
The program described in Appendix 13 or 14,
The biological information indicated by the time-series data is information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, information indicating the direction of the face, information indicating the pulse wave, information indicating the blood flow, and information of the mouth of the user. At least one of the information indicating the degree of opening / closing,
A program characterized by that.

(Appendix 16)
The program described in Appendix 15
When the biometric information indicated by the time series data is two or more pieces of information, the learning model has a layer for convolution for each of the biometric information.
A program characterized by that.

(Appendix 17)
The program described in any of Appendix 13 to 16.
Before Symbol computer,
(E) in response to the estimated the awakening degree, to adjust the frame rate, Ru further to execute the steps,
A program characterized by that.

(Appendix 18)
The program described in Appendix 17,
In the step (e), the frame rate is further adjusted according to the biometric information indicated by the extracted time series data.
A program characterized by that.

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the structure and details of the present invention.

As described above, according to the present invention, it is possible to accurately estimate the arousal level of a person while reducing the processing load. The present invention is useful for various systems that require estimation of human alertness, such as air conditioning systems, vehicle operation systems such as automobiles, and the like.

10 Alertness estimation device (Embodiment 1)
11 Image data acquisition unit 12 Time-series data extraction unit 13 Data processing unit 14 Arousal level estimation unit 20 Imaging device 30 Arousal level estimation device (Embodiment 2)
31 Frame rate adjustment unit 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (18)

A device for estimating the user's alertness,
An image data acquisition unit that acquires image data including the user's face image at a set frame rate, and
A time-series data extraction unit that extracts time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and a time-series data extraction unit.
A data processing unit that interpolates the time-series data so that the number of samples of the extracted time-series data becomes a set value.
An arousal level estimation unit that estimates the arousal level of the user by inputting the interpolated time series data into a learning model constructed using a convolutional neural network.
An arousal level estimation device characterized by being equipped with. The arousal level estimation device according to claim 1.
The training model is constructed by inputting data obtained by interpolating a plurality of time series data having different frame rates so that the number of samples becomes a set value into a convolutional neural network as training data. Has been
An arousal level estimation device characterized by this. The alertness estimation device according to claim 1 or 2.
The biological information indicated by the time-series data includes information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, information indicating the direction of the face, information indicating pulse waves, information indicating blood flow, and information of the mouth of the user. At least one of the information indicating the degree of opening / closing,
An arousal level estimation device characterized by this. The alertness estimation device according to claim 3.
When the biometric information indicated by the time series data is two or more pieces of information, the learning model has a layer for convolution for each of the biometric information.
An arousal level estimation device characterized by this. The alertness estimation device according to any one of claims 1 to 4.
It further includes a frame rate adjusting unit that adjusts the frame rate according to the estimated alertness.
An arousal level estimation device characterized by this. The alertness estimation device according to claim 5.
The frame rate adjusting unit further adjusts the frame rate according to the biometric information indicated by the extracted time series data.
An arousal level estimation device characterized by this. A method for estimating user alertness,
(A) A step of acquiring image data including the user's face image at a set frame rate, and
(B) A step of extracting time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and
(C) A step of interpolating the time-series data so that the number of samples of the extracted time-series data becomes a set value.
(D) A step of inputting the interpolated time series data into a learning model constructed by using a convolutional neural network to estimate the arousal degree of the user.
A method for estimating arousal level, which comprises. The arousal level estimation method according to claim 7.
The training model is constructed by inputting data obtained by interpolating a plurality of time series data having different frame rates so that the number of samples becomes a set value into a convolutional neural network as training data. Has been
An alertness estimation method characterized by this. The arousal level estimation method according to claim 7 or 8.
The biological information indicated by the time-series data includes information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, information indicating the direction of the face, information indicating pulse waves, information indicating blood flow, and information of the mouth of the user. At least one of the information indicating the degree of opening / closing,
An alertness estimation method characterized by this. The arousal level estimation method according to claim 9.
When the biometric information indicated by the time series data is two or more pieces of information, the learning model has a layer for convolution for each of the biometric information.
An alertness estimation method characterized by this. The arousal level estimation method according to any one of claims 7 to 10.
(E) Further having a step of adjusting the frame rate according to the estimated alertness.
An alertness estimation method characterized by this. The arousal level estimation method according to claim 11.
In the step (e), the frame rate is further adjusted according to the biometric information indicated by the extracted time series data.
An alertness estimation method characterized by this. A program for estimating the arousal level of the user by a computer,
On the computer
(A) A step of acquiring image data including the user's face image at a set frame rate, and
(B) A step of extracting time-series data indicating the biometric information of the user from the image data acquired at the set frame rate, and
(C) A step of interpolating the time-series data so that the number of samples of the extracted time-series data becomes a set value.
(D) A step of inputting the interpolated time series data into a learning model constructed by using a convolutional neural network to estimate the arousal degree of the user.
Ru is the execution, program. The program according to claim 13.
The training model is constructed by inputting data obtained by interpolating a plurality of time series data having different frame rates so that the number of samples becomes a set value into a convolutional neural network as training data. Has been
A program characterized by that. The program according to claim 13 or 14.
The biological information indicated by the time-series data includes information indicating the degree of opening and closing of the eyes, information indicating the direction of the line of sight, information indicating the direction of the face, information indicating pulse waves, information indicating blood flow, and information of the mouth of the user. At least one of the information indicating the degree of opening / closing,
A program characterized by that. The program according to claim 15.
When the biometric information indicated by the time series data is two or more pieces of information, the learning model has a layer for convolution for each of the biometric information.
A program characterized by that. The program according to any one of claims 13 to 16.
On the computer
(E) in response to the estimated the awakening degree, to adjust the frame rate, Ru further to execute the steps,
A program characterized by that. The program according to claim 17.
In the step (e), the frame rate is further adjusted according to the biometric information indicated by the extracted time series data.
A program characterized by that.

Images