JP6090653B2 - Fatigue determination device, fatigue determination method, and program thereof - Google Patents

Description

  The present invention relates to an apparatus, a method, and a program for determining a person's fatigue level based on moving image data including the person.

  Conventionally, for example, various methods have been studied as a method of determining the fatigue level of a person such as a driver in a long-distance driving of a passenger car or a truck or an auditor during a lecture.

  As a method for predicting human drowsiness, it has been proposed to measure and evaluate blinks of the driver's eyes, brain waves, heartbeat, respiration, and the like (Non-Patent Document 1).

  In addition, it has been proposed to perform face recognition using image data obtained by photographing a driver and perform feature point analysis to estimate a decrease in arousal level (Non-Patent Document 2).

Hiroki Kitajima, Nakaho Numata, Keiichi Yamamoto, Misa Goi, “Study on prediction method of sleepiness when driving a car: 1st report, Evaluation method of sleepiness expression and effective index for prediction of sleepiness fluctuation” Japan Society of Mechanical Engineers Proceedings C, Vol.63, No.613-C, pp.93-100 (1997) Kenji Ishida, Chiri Hachisuka, “Driver State Sensing Using Facial Expressions: Approach and Future Prospects for Detection of Awakening Decline”, Automobile Engineering Society Symposium, 02-11, pp.12-17, November 2011 11th

  In the method according to Non-Patent Document 1, it is necessary to measure an electroencephalogram, a heartbeat, respiration, and the like using individual sensors, and there is a problem that the device becomes expensive, and it is always necessary for a driver to wear many sensors. It was difficult in reality. In addition, the method according to Non-Patent Document 2 has a problem in that an algorithm suitable for each user is required to estimate the arousal level because the feature points differ for each photographed user.

  The present invention relates to a fatigue level determination apparatus that determines a fatigue level based on a luminance change per unit time of moving image data including a person.

  The present invention compares an image acquisition unit that acquires images per unit time with moving image data obtained by photographing a person, and an image that calculates pixel change data between adjacent images by comparing images acquired per unit time The comparison unit and the pixel change data in the first predetermined period of the moving image data are used as reference data, and the fatigue level in the second predetermined period is determined based on the reference data and the pixel change data in the second predetermined period. A fatigue determination device having a fatigue determination unit.

  In addition, the fatigue determination unit of the fatigue determination apparatus of the present invention calculates a variation value by calculating the number of times the pixel change data in the first and second predetermined periods has changed more than a predetermined value, You may determine the fatigue degree in a predetermined period.

  The fatigue determination apparatus may further include a reference data storage unit that stores reference data, and the fatigue determination unit may read the reference data from the reference data storage unit and determine the degree of fatigue.

  In the fatigue determination device, the fatigue determination unit performs a t-test between the variation value of the reference data and the variation value at the second predetermined time, and determines that the fatigue is in the second predetermined period if significantly different. You may make it do.

  In the fatigue determination apparatus, the image comparison unit may detect a luminance change as pixel change data.

  In the fatigue determination apparatus, the image acquisition unit may determine an image range to be acquired in advance, and the image comparison unit may calculate pixel change data in the determined image range among the image data.

The present invention includes a step of acquiring an image every unit time in moving image data obtained by photographing a person, a step of comparing pixel images acquired every unit time, and calculating pixel change data between adjacent images, Based on the reference data and the pixel change data in the second predetermined period, the pixel change data in the first predetermined period is a predetermined value based on the reference data and the pixel change data in the second predetermined period. The present invention provides a fatigue determination method comprising: calculating a variation value by calculating the number of times of change as described above and determining a fatigue level in the second predetermined period.

A step of acquiring an image every unit time in moving image data in which a person is photographed, a step of comparing pixel images acquired every unit time and calculating pixel change data between adjacent images, a first of the moving image data Pixel change data for a predetermined period of time is used as reference data, and based on the reference data and the pixel change data for the second predetermined period, the pixel change data for the first and second predetermined periods is changed to a predetermined value or more. A fatigue determination program for calculating a fluctuation value by calculating the number of times of occurrence and causing a computer to execute a step of determining a fatigue level in the second predetermined period is provided.

  According to the present invention, it is possible to determine the fatigue level without using a special sensor or the like because the fatigue level is determined by extracting a change in pixels per unit time of moving image data including a person. Become. Moreover, in order to determine the degree of fatigue by extracting the change of the pixel in the image data for the minute human movement that increases with fatigue, a simple device configuration and without using a complex algorithm such as face recognition It becomes possible to determine the degree of fatigue by an algorithm.

It is a block diagram of the fatigue determination apparatus in one embodiment of the present invention. It is a block diagram which shows an example of the hardware constitutions of the fatigue determination apparatus 100 which concerns on embodiment of this invention. It is a flowchart which shows an example of the process which calculates pixel change data from moving image data. It is a figure explaining an example of the determination method of the image range acquired in step S4. 5 is a flowchart illustrating an example of a process for calculating a variation value from pixel change transition data in a fatigue determination unit 105; 5 is a flowchart illustrating an example of fatigue determination processing in a fatigue determination unit 105.

  Hereinafter, a fatigue determination apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of a fatigue determination apparatus in one embodiment of the present invention. The fatigue determination apparatus 100 is an apparatus that determines the degree of fatigue of a person based on moving image data obtained by imaging the person. For example, the fatigue determination apparatus 100 acquires an image for each unit time based on moving image data input from a camera that captures a driver's seat of an automobile, and measures a pixel change between adjacent images, for example, a luminance change. The degree of fatigue is determined by comparing the fluctuation value of the luminance change in a certain period with, for example, the fluctuation value in a predetermined period at the beginning of operation. In addition to fatigue determination of a person driving a car, fatigue determination is possible in other cases. For example, fatigue determination may be performed based on moving image data captured by a camera set to capture a seat of a student taking a class.

  The fatigue determination apparatus 100 is connected to the imaging unit 200 and the output unit 300, for example. The imaging unit 200 is an apparatus for capturing moving image data including a person, such as a video camera. When the moving image data captured by the imaging unit 200 is input, the fatigue determination device 100 performs fatigue determination based on the moving image data. Note that it is practically desirable to process moving image data captured by the imaging unit 200 in real time. However, fatigue determination may be performed by temporarily storing moving image data that has been captured in advance and reading the stored moving image data.

  The output unit 300 outputs the result of fatigue determination in the fatigue determination apparatus 100. For example, a sound output device such as a display device or a microphone that outputs a warning sound or an announcement is applicable.

  The fatigue determination apparatus 100 includes an input unit 101, an image acquisition unit 102, an image comparison unit 103, a reference data storage unit 104, and a fatigue measurement unit 105. For example, the input unit 101 inputs moving image data captured by the imaging unit 200. The moving image data stored in the external storage unit may be input without being limited to the moving image data captured by the imaging unit.

  The image acquisition unit 102 acquires image data from the moving image data input from the input unit 101 every unit time. The unit time is a sampling time. For example, when the sampling time is set to 0.5 seconds, the unit time is 0.5 seconds, and the image acquisition unit 102 acquires image data every 0.5 seconds. The image acquisition unit 102 may acquire image data for each unit time only in the moving image data of a predetermined period among the input moving image data. The predetermined period is a target period for determining the degree of fatigue, for example. For example, image data may be acquired for each unit time only for the first 10 minutes and the last 10 minutes of the input moving image data, and other periods may not be acquired. The image acquisition unit 102 acquires image data for each unit time in at least two predetermined periods of moving image data, one is a period for reference data, and the other is a period for fatigue determination. .

  For the image data acquired by the image acquisition unit 102, the image comparison unit 103 sets pixel values at corresponding coordinates between adjacent images, that is, between the acquired image data and the image data of the next unit time after the acquired image data. The pixel change amount is calculated by comparison. Then, pixel change data based on the pixel change amount is obtained. For example, the image comparison unit 103 calculates pixel change data and outputs it as pixel change transition data in all the image data acquired in a specified predetermined period. Note that the image comparison unit 103 may determine a pixel comparison range in the image data in advance and calculate the pixel change amount only in that range.

  The reference data storage unit 104 stores data based on pixel changes in a period serving as a reference for moving image data. The reference period is, for example, the first predetermined period when shooting of a person is started in moving image data. Although the reference period is not limited to this, it is for determining the degree of fatigue, so it is desirable that the period is not fatigued. For example, it is a time when driving or work is started or a period close thereto. The reference data based on the pixel change is, for example, pixel change transition data between adjacent images for each unit time within a predetermined period. It should be noted that the number of times of change from a pixel change amount in a reference period to a predetermined value or more may be calculated as a variation value and stored as reference data.

  The fatigue determination unit 105 determines the degree of fatigue in the predetermined period based on the reference data and the pixel change data in a predetermined period. For example, the fatigue level is determined by comparing the variation value of the pixel change in the reference data with the variation value of the pixel change in the fatigue determination target period. Note that the fatigue level may be determined by performing t-test on the variation value. A specific determination method will be described later.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the fatigue determination apparatus 100 according to the embodiment of the present invention. In FIG. 2, the computer constituting the fatigue determination apparatus 100 can be realized by a general-purpose hardware configuration that has existed in the past. That is, the computer forming the fatigue determination device 100 connects the keyboard 108 connected to the CPU 101, ROM 102, RAM 103, external storage device 104, communication interface 105, and input / output interface 106 to the bus as shown in FIG. Composed. Further, as an output unit 300, for example, a speaker 301 and a display 302 provided as a display device are connected to the input / output interface 106.

  In a case where a series of processing is executed by software, for example, the image acquisition unit 102, the image comparison unit 103, and the fatigue determination unit 105 include a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory as the external storage device 104. Various functions are executed by being stored in a removable medium as a fatigue determination program and executed by the CPU 101. Note that the fatigue determination program may be installed from the network to the computer and may be configured in a state of being pre-installed in the apparatus main body, for example, the ROM 102 in which the program is recorded.

  FIG. 3 is a flowchart illustrating an example of processing for calculating a pixel change amount from moving image data. The image acquisition unit 102 reads the moving image data input from the input unit 101 (step S1). Next, the image acquisition unit 102 acquires the final frame number (step S2). Here, the final frame number is the frame number of the final image in the period in which the pixel change is acquired in the moving image data. For example, in the moving image data, if the predetermined period serving as a reference and the target period for determining the degree of fatigue are 10 minutes, the frame number of the last image in 10 minutes is acquired.

  Next, the image acquisition unit 102 determines a unit time (sampling time) (step S3). Here, the time interval in the case of acquiring an image for every unit time is determined. For example, the image acquisition unit 102 determines the sampling time based on a frame rate of moving image data and a period for acquiring a pixel change amount. For example, the image acquisition unit 102 may store a sampling time table corresponding to the frame rate and determine the sampling time based on the table. Similarly, a table storing the sampling time corresponding to the period for acquiring the pixel change amount may be stored and determined based on the table. If the sampling time is determined in advance, this step S3 and the step S2 for obtaining the final frame number are unnecessary. Here, as an example, the reference period and the period for acquiring the pixel change amount are 10 minutes, and the unit time is 0.5 seconds.

  The image acquisition unit 102 determines an image range to be acquired (step S4). In the frame image data acquired from the moving image data, an image range for calculating the pixel change amount is determined. An image range to be acquired is determined so as to remove an area in which pixels other than a person appear and change in pixels, such as a background portion in the captured moving image data. As described above, the image acquisition unit 102 determines in advance by designating an image range in which the pixel change amount is acquired in the image data acquired every unit time by the coordinate data or the like.

  Note that steps S2 to S4 can be omitted if they have already been determined as specified values. In that case, the process proceeds to step S5 after step S1.

  The image acquisition unit 102 acquires an initial frame image (step S5). The image acquisition unit 102 acquires the image of the first frame in the period determined to acquire the pixel change amount. The image acquisition unit 102 sets the initial image as the current image (step S6), and acquires an image one unit time ahead (step S7).

The image comparison unit 103 compares between the adjacent images of the current image acquired by the image acquisition unit 102 and the image one unit time ahead (step S8). As an example, pixel values at the same coordinates are compared between adjacent images for each pixel. The pixel value is, for example, a luminance value. Not only the luminance value but also RGB values may be used. When calculating the pixel change amount using the RGB values, for example, the image comparison unit 103 calculates the pixel change amount (PT) by the following method.

  Here, i is a coordinate value in the x coordinate of a pixel (pixel) on the image, and j is a coordinate value in the y coordinate of a pixel (pixel) on the image. A change amount of pixels having the same coordinate value is calculated for each pixel between adjacent images, and a pixel change amount (PT) is calculated by summing up absolute values of pixel differences. Further, pixel change data is calculated by dividing the pixel change amount by the total number of pixels to obtain an average value. Here, the average value is used, but the pixel change amount at a specific coordinate may be used as the pixel change data, or the pixel change amount itself of the entire image data may be used as the pixel change data. If the image range for calculating the pixel change data is determined in step S4, the pixel change data is calculated only in the determined range in the image.

  The image comparison unit 103 stores pixel change data (step S9).

  Next, the image acquisition unit 102 sets an image one unit time ahead as the current image (step S10). This is for comparison with the next unit time ahead image. The image acquisition unit 102 determines whether the current image is the final frame image (step S11). If the current image is not the final frame image (No), the image acquisition unit 102 acquires the next unit time ahead image, The comparison unit 103 compares pixel values. When it is the last frame image of the predetermined period (Yes), the image comparison unit 103 outputs the pixel change data for the predetermined period to the file as the pixel change transition data (step S12). The output transition data of the pixel change amount during a predetermined period may be stored in a storage unit (not shown) or may be output to the fatigue determination unit 105 as it is. Note that the image comparison unit 103 stores pixel change transition data in the reference period in the reference data storage unit 104. However, the present invention is not limited to this, and the measured data of the output pixel change transition data may be stored in the reference data storage unit 104 regardless of the reference period or the determination period.

  FIG. 4 is a diagram illustrating an example of a method for determining the image range to be acquired in step S4. FIG. 4A shows a region where a luminance change caused by a person can not occur and a region where the luminance change can occur in a moving image captured by an imaging unit such as a camera by hatching. Yes. Here, a person sitting in the driver's seat of the car is photographed. The scenery seen from the window is an area where the luminance can change, and the portion behind the seat is hatched as an excluded area because there is no possibility of a person appearing. As shown in FIG. 4B, the hatched portion is removed, and the range is specified so that only the area surrounded by the thick line is calculated. The range is specified by a method such as specifying coordinates in the image. Then, as shown in FIG. 4C, only the image range in which the pixel change amount is acquired is set, and the value of each pixel is calculated. Such determination of the area may be performed individually on the moving image data.

  FIG. 5 is a flowchart illustrating an example of processing for calculating a variation value from pixel change transition data in the fatigue determination unit 105. The fatigue determination unit 105 reads pixel change transition data for a predetermined period output from the image comparison unit 103 or pixel change transition data read from the storage unit (step S13).

数式におけるｎは、所定期間内の画素変化推移データの要素数である。 Next, the fatigue determination part 105 calculates the threshold value for fatigue degree determination (step S14). For example, the average value of the pixel change data is calculated for the pixel change transition data in the target period, and the average value (Mean) plus 1SD (Standard Deviation standard deviation) is calculated as the threshold value. However, the present invention is not limited to this, and the threshold value may be determined by other methods. The determined threshold value is used to count how many times the deviation has occurred from the average and how many pixel changes have occurred. That is, the degree to which the person who is subject to fatigue determination moves beyond the average than the normal movement is measured. For example, the average value and the standard deviation are calculated by the following equations.

N in the formula is the number of elements of the pixel change transition data within a predetermined period.

  The fatigue determination unit 105 classifies the pixel change transition data based on the determined reference value (step S15). For example, the total number of times that the pixel change data has increased or decreased by 1 SD or more from the average value is calculated as the fluctuation value for the predetermined period. The fluctuation value is calculated according to this flowchart for both the reference period and the determination target period.

  The fatigue determination unit 105 outputs the calculated fluctuation value to a file (step S16). The calculated variation value may be temporarily stored in a memory or the like. For example, the variation value in the reference period may be stored in the reference data storage unit 104.

  FIG. 6 is a flowchart illustrating an example of fatigue determination processing in the fatigue determination unit 105. The fatigue determination unit 105 reads the fluctuation value in the reference period and the fluctuation value in the fatigue determination target period (step S17). For example, the fluctuation value in the reference period is read from the reference data storage unit 104. Moreover, when performing determination in real time about an object period, it transfers to step S17 as it is from step S16.

  The read fluctuation value in the reference period and the fluctuation value in the fatigue determination target period are t-tested (step S18). For example, when the p value in the t test is less than 0.05, it is determined that the user is tired. Here, the t-test is used for comparison, but other methods such as a non-parametric method (such as a code rank test) may be used.

  Next, the fatigue determination unit 105 performs fatigue determination (step S19). For example, if the results of t-test are significantly different, it is determined that the subject is in a fatigued state. Notification may be performed by outputting the determination result to an output unit 300 such as a display device such as a display or an audio output unit such as a microphone. When performing t-test, the warning weight may be changed according to the p-value in the t-test.

  As described above, the degree of fatigue can be determined by measuring the minute change in the movement of the person accompanying fatigue by calculating the pixel change. In the above description, the example of photographing the driving person has been described. However, the present invention is not limited to this, and the degree of fatigue of these persons can also be determined by photographing a student who is in class or a person who is working.

Claims (7)

An image acquisition unit that acquires an image per unit time in moving image data in which a person is photographed;
An image comparison unit that compares images acquired every unit time and calculates pixel change data between adjacent images;
The pixel change data in the first predetermined period of the moving image data is used as reference data, and the pixel change in the first and second predetermined periods is based on the reference data and the pixel change data in the second predetermined period. A fatigue determination device comprising: a fatigue determination unit that calculates a variation value by calculating the number of times of change from a data to a predetermined value or more and determines a fatigue level in the second predetermined period. A reference data storage unit for storing the reference data;
The fatigue determination unit, the fatigue determining device reference data according to claim 1, wherein determining the degree of fatigue is read from the reference data storage unit. The fatigue determination unit, and a variation in the variation value and a second predetermined time reference data t test, if significantly different, the second and judges claim 1 or fatigued in a predetermined time period 2. The fatigue determination apparatus according to 2 . The image comparison section, the fatigue determining device according to claim 1-3, wherein the detecting a change in luminance as the pixel variation data. Wherein the image acquiring unit predetermine the image area to be acquired, the image comparison section, the fatigue determining device according to claim 1-4, wherein calculating the pixel change data in the image range determined in the image data. Acquiring an image every unit time in moving image data in which a person is photographed;
Comparing the images acquired every unit time and calculating pixel change data between adjacent images;
The pixel change data in the first predetermined period of the moving image data is used as reference data, and the pixel change in the first and second predetermined periods is based on the reference data and the pixel change data in the second predetermined period. Calculating a fluctuation value by calculating the number of times the data has changed to a predetermined value or more from the data, and determining a fatigue level in the second predetermined period;
And a method for determining fatigue. Acquiring an image every unit time in moving image data in which a person is photographed;
Comparing the images acquired every unit time and calculating pixel change data between adjacent images;
The pixel change data in the first predetermined period of the moving image data is used as reference data, and the pixel change in the first and second predetermined periods is based on the reference data and the pixel change data in the second predetermined period. Calculating a fluctuation value by calculating the number of times the data has changed to a predetermined value or more from the data, and determining a fatigue level in the second predetermined period;
Fatigue judgment program to make computer execute.

Images