JP5365374B2 - Deficiency determination apparatus and method - Google Patents

Description

  The present invention relates to a deficiency determination device and method for determining a human deficit state.

  2. Description of the Related Art Conventionally, a technique described in Patent Document 1 below is known as a state monitoring device that determines the behavior of a human mouth.

  In Patent Document 1, the size of the entire mouth when opening in the vertical and horizontal directions is measured, and a differential value of the temporal change in the size of the opening is obtained. Then, the time is divided at the inflection point of the differential value, and it is determined whether the state is in a stretched state or a conversation state based on the number of segments in the divided time series and the length of time of each segment.

JP-A-2005-199078

  However, in the technique described in Patent Document 1 described above, the behavior of the mouth portion is determined by pattern matching using the differential value of the temporal change in the opening size. For this reason, there is a problem that the misjudgment rate becomes high depending on how the predetermined pattern is set (particularly, paragraph 0082 of Patent Document 1).

  Therefore, the present invention has been proposed in view of the above-described circumstances, and an object thereof is to provide an elongation determination apparatus and method that can perform determination of an elongation state with high accuracy without using pattern matching. .

  In order to solve the above-described problem, the present invention calculates the aspect ratio of the mouth based on the feature points extracted from the mouth image included in the face image, determines the open / closed state of the mouth, and enters the open state. It is determined whether or not the stretched state is based on the open time.

  According to the present invention, the opening / closing state of the mouth is calculated by calculating the aspect ratio of the mouth, and it is determined whether or not the stretched state is based on the opening time in the open state. It is possible to determine the state, and it is possible to determine the stretched state with high accuracy.

It is a block diagram which shows the outline of the lack extension determination system which shows this invention as embodiment. It is a figure explaining the example of installation of a camera system in the lack extension judging system shown as an embodiment of the present invention. It is a block diagram which shows the functional structure of the deficit calculating apparatus in the deficient determination system shown as embodiment of this invention. It is a figure explaining the process which extracts the feature point from a face image, and calculates | requires height and width in the lack extension determination system shown as embodiment of this invention. In the non-expansion determination system shown as an embodiment of the present invention, (a) shows the time change of the aspect ratio, and (b) shows the time change of the mouth open / close state. In the non-expansion determination system shown as an embodiment of the present invention, (a) shows the time change of the aspect ratio, and (b) to (d) show the time change of the mouth open / close state. It is a figure which shows the relationship between an open determination threshold value and a period threshold value in the non-expansion determination system shown as embodiment of this invention. It is a flowchart which shows the process sequence which calculates | requires an open determination threshold value in the non-expansion determination system shown as embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The stretch detection system shown as an embodiment of the present invention is configured, for example, as shown in FIG. The defect determination system includes a defect calculation apparatus 1 that performs various calculations for determining defect, a camera device (imaging means) 2 that acquires a face image, and an infrared light source 3 that irradiates an area including a driver's face with infrared rays. With. This lack determination system captures a face image by the camera device 2 while the infrared light source 3 irradiates infrared rays. Of course, the camera device 2 is not limited to an infrared camera, and may be a camera of another type.

  The failure determination system outputs a failure detection signal when the failure calculation device 1 detects the driver's failure state. This lack detection signal is supplied to, for example, a dozing detection device or a dozing prevention device (not shown). Thereby, the lack detection signal can be used, for example, to output an alarm sound for preventing dozing from a speaker or the like.

  For example, as shown in FIG. 2, the lack determination system is provided in a driver seat operated by a driver. The camera system 102 including the camera device 2 and the infrared light source 3 is installed in the vicinity of the steering 100 and in front of the instrument 101 that can be visually recognized by the driver. Thereby, the camera system 102 can photograph a face image including at least the mouth portion of the driver. A face image obtained by imaging with the camera device 2 is supplied to the non-expansion calculation device 1.

  The non-expansion calculation device 1 includes functional units as shown in FIG. 3, for example. Note that although the non-expansion arithmetic unit 1 is actually composed of a ROM, a RAM, a CPU, and the like, functions that can be realized by the CPU performing a process according to a program for determining an expansion is stored as a block. explain.

  The stretch calculation device 1 includes a face feature detection unit 11, a feature selection / measurement unit 12, an aperture ratio calculation unit 13, an open determination unit 14, a stretch detection unit 15, an open determination threshold update unit 16, and a period threshold detection unit. 17 is included.

The face image detection unit 11 is supplied with the camera image data acquired by the camera device 2. The face feature detection unit 11 extracts feature points from the mouth image of the face images included in the camera image data (feature point extraction means). For example, as shown in FIG. 4, the face feature detection unit 11 extracts a feature point m of the mouth image 21 from the face image 20. The feature point m includes a feature point m _{ul at} the upper end point, a feature point m _{lc at} the right end point, a feature point m _{rc at} the left end point, and a feature point m _{ll at the} lower end point.

The feature selection / measurement unit 12 calculates the aspect ratio of the mouth based on the feature point m extracted by the face feature detection unit 11 (aspect ratio calculation means). For example, as shown in FIG. 4, the feature selection / measurement unit 12 has a height H that is a vertical distance between the feature point m _ul of the upper end point and the feature point m _ll of the lower end point, and the feature point m _{lc of the} right end point A width W that is a lateral distance from the feature point m _rc at the left end point is calculated.

  The opening ratio calculation unit 13 calculates an aspect ratio that is a ratio between the height H and the width W calculated by the feature selection / measurement unit 12.

  The open determination threshold value update unit 16 calculates a predetermined open determination threshold value in advance. The open determination threshold value is determined based on the aspect ratio calculated by the opening ratio calculation unit 13. The open determination threshold value is an aspect ratio value that allows the driver to distinguish whether he / she is missing or talking. This open determination threshold value may be a predetermined value or a value updated depending on how the driver opens. The opening determination threshold value update process will be described later.

  The open determination unit 14 determines the opening / closing state of the mouth based on the aspect ratio calculated by the opening ratio calculation unit 13 (mouth opening / closing state determination unit). At this time, the open determination unit 14 determines the open state or the closed state by comparing the open determination threshold updated by the open determination threshold update unit 16 with the aspect ratio. Moreover, the open determination part 14 calculates the opening time which is the said open state, when it determines with the opening / closing state of the present opening | mouth being an open state (opening time calculation means).

  The period threshold detection unit 17 detects the period threshold based on the open determination threshold updated by the open determination threshold update unit 16. The period threshold is a period in which it is possible to distinguish whether the driver is absent or talking. This period threshold value may be a predetermined value or a value updated depending on how the driver opens. The period threshold value update process will be described later.

  When the opening time in the open state determined by the opening determination unit 14 is sufficiently long, the defect detection unit 15 detects that the defect has occurred and outputs a defect detection signal. Note that the stretch detection unit 15 determines whether the stretch is in a stretched state or is simply opened using a period threshold value λ described later.

  Next, the calculation process of the opening time in the open state in the above-described failure extension determination system will be described.

  In the non-expansion determination system, the aspect ratio ρ is calculated by the opening ratio calculation unit 13 as time changes. Here, the aspect ratio ρ is height H / width W, and increases as the height H increases. Therefore, the higher the height H and the larger the aspect ratio ρ, the easier it is to determine that the mouth is open.

  For example, assume that the aspect ratio ρ has changed as shown in FIG. At this time, the open determination unit 14 compares the open determination threshold τ set in advance by the open determination threshold update unit 16 with the aspect ratio ρ, and the aspect ratio ρ satisfies the open determination threshold τ. It is possible to detect the timing when it exceeds and the timing when the aspect ratio ρ falls below the open determination threshold value τ. Then, in FIG. 5A, the open determination unit 14 can detect a pair of t1 and t2, t3 and t4, and t5 and t6.

In this change in the aspect ratio ρ, the open determination unit 14 starts a timer at time t1 when the aspect ratio ρ exceeds the open determination threshold τ, as shown in FIG. At this time, the open determination unit 14 sets a flag “1” indicating that the mouth open / close state is the open state. Thereafter, the open determination unit 14 stops the timer at time t2 when the aspect ratio ρ falls below the open determination threshold value τ. At this time, the open determination unit 14 returns the flag “1” in which the mouth open / close state is the open state to “0”. Thereby, the open determination part 14 can acquire the opening time _dt1 which is in an open state at the time t2. Similarly, the open determination unit 14 can acquire the opening time d _{t2 in} the open state from t3 to t4 and the opening time d _t3 in the open state from t5 to t6.

  Next, the update processing of the open determination threshold value τ and the period threshold value λ in the above-described failure extension determination system will be described with reference to the processing explanatory diagrams of FIGS. 6 and 7 and the flowchart of FIG.

  As shown in FIG. 8, first, the open determination threshold value update unit 16 collects the latest n number of aspect ratios ρ in step S1. This n aspect ratio ρ is performed for each predetermined detection window T, and the process proceeds to the following process.

  In the next step S2, the open determination threshold value updating unit 16 creates a histogram H using the aspect ratio ρ collected in step S1.

  For example, as shown in FIG. 6A, the lack determination system calculates the aspect ratio ρ by the opening ratio calculation unit 13 as time changes. The open determination threshold value updating unit 16 collects the aspect ratio ρ for each predetermined detection window T and creates a histogram H while calculating the time change of the aspect ratio ρ. The width (period) of the predetermined detection window T is a period during which the open determination threshold value τ and the period threshold value λ can be set with high accuracy, and is set in advance by statistical analysis and stored in a memory (not shown). The width of the predetermined detection window T is, for example, a period of a few minutes. 6B, 6C, and 6D, the horizontal axis represents the aspect ratio ρ, and the vertical axis represents the frequency of the aspect ratio ρ detected during the period of the window T. FIGS. 6B, 6C, and 6D show histograms created in different cases.

Returning to FIG. 8, in the next step S <b> 3, the open determination threshold value update unit 16 counts the aspect ratio ρ of the histogram H created in step S <b> 2 from the upper value, and within the predetermined detection window T. The region up to β% of the sampled total aspect ratio ρ is determined. As a result, as shown in FIGS. 6B to 6C, it is possible to determine the region that is β% of the total aspect ratio ρ, and to determine the aspect ratio ρ that is the lowest value of the region (β%). . The aspect ratio ρ that is the lowest value is the open determination threshold value τ _t . Here, as shown in FIGS. 6B to 6C, the histograms H have different shapes depending on the driver, and therefore the open determination threshold τ is different for each histogram H. Therefore, it is possible to obtain the open determination threshold value τ suitable for the behavior of opening the driver's mouth and the method of the extension.

In the next step S4, the open determination threshold value updating unit 16 determines the upper and lower limit ranges τ _min and τ _max of the open determination threshold value obtained in step S3. This upper / lower limit range is a range of the aspect ratio ρ in which there is a high possibility that the driver's mouth is in the open state, and is determined by statistical data in advance.

In the next step S5, the open determination threshold value updating unit 16 determines that the open determination threshold value τ _t obtained in step S3 is between the upper and lower limit ranges τ _min and τ _max determined in step S4. It is determined whether or not it is included. If the open determination threshold value τ _t is included between the upper and lower limit ranges τ _min and τ _max , the process proceeds to step S6, and if not, the process proceeds to step S7.

In step S6, the open determination threshold value update unit 16 outputs the open determination threshold value τ _t obtained in step S3 to the open determination unit 14 and the period threshold value detection unit 17, and ends the process. .

On the other hand, in step S7, the open determination threshold value updating unit 16 determines the open determination threshold value τ _t to be output as one of the upper and lower limit ranges τ _min and τ _max . When the open determination threshold τ _t is smaller than τ _{min in the} upper and lower limit range, τ _min is output as the open determination threshold τ _t . Conversely, when the open determination threshold value τ _t is larger than τ _{max in the} upper and lower limit range, τ _max is output as the open determination threshold value τ _t .

By such determination processing of the open determination threshold value τ _t , the open determination unit 14 can determine the open / closed state of the mouth using the open determination threshold value τ _t .

Moreover, the period threshold value detection part 17 determines period threshold value (lambda) using the open determination threshold value (tau) _t . As shown in FIG. 7, the period threshold value detection unit 17 stores a comparison table indicating the relationship between the open determination threshold value τ _t and the period threshold value λ in advance. The period threshold value detection unit 17 reads the period threshold value λ _t corresponding to the opening determination threshold value τ _t determined by the opening determination threshold value update unit 16 from the comparison table. As a result, the period threshold value detection unit 17 determines the period threshold value λ and outputs it to the deficit detection unit 15. In this comparison table, even if any of the upper and lower limit ranges τ _min and τ _max is output from the open determination threshold update unit 16, the period threshold value detection unit 17 does not change the upper and lower limit ranges τ _min and τ _max. It is possible to determine the period thresholds λ _min and λ _max in the upper and lower limit range corresponding to.

  As described above, according to the lack determination system shown as the embodiment of the present invention, the feature point is extracted from the mouth image included in the face image, the aspect ratio of the mouth is calculated, and the mouth aspect ratio is calculated based on the aspect ratio. The open / close state of is determined. And based on the opening time which is in an open state, it is determined whether it is a stretched state. Thereby, according to the absence extension determination system, the absence extension determination can be performed without performing pattern matching for the mouth image and the opening / closing state of the mouth. Therefore, according to this failure determination system, it is possible to avoid a decrease in accuracy of the failure determination due to pattern matching, and to determine the failure with high accuracy.

  In addition, according to this non-expansion determination system, when the aspect ratio is larger than a predetermined open determination threshold value, it can be determined as an open state. Further, according to this non-extension determination system, it is possible to determine that the state is the extension state when the opening time is longer than a predetermined period threshold value.

  Further, according to this lack determination system, the aspect ratio is collected during a predetermined period, a histogram is created, the aspect ratio of the histogram is counted from the upper value, and the aspect ratio at a predetermined ratio of the entire histogram is opened. Determined as determination threshold. Thereby, the open determination threshold can be updated, and the open determination threshold for accurately determining the lack of extension can be determined for each driver.

  Furthermore, according to this non-expansion determination system, the updated open determination threshold value is compared with a predetermined range that can be determined as an open state based on the aspect ratio, and the determined open determination threshold value is lower than the predetermined range. In this case, the minimum value of the predetermined range is determined as the open determination threshold value, and when the determined open determination threshold value is higher than the predetermined range, the maximum value of the predetermined range is set as the open determination threshold value. decide. Thereby, according to the lack determination system, even when the open determination threshold updated using the histogram becomes a very large value, the open determination threshold can be determined within a predetermined range. Decrease can be avoided.

  Furthermore, according to the lack determination system, a predetermined period threshold value is updated based on the open determination threshold value. Therefore, a period threshold value corresponding to the open determination threshold value set for each driver is set. The threshold value for determining the lack of accuracy can be determined accurately for each driver.

DESCRIPTION OF SYMBOLS 1 Decompression calculation apparatus 2 Camera apparatus 3 Infrared light source 11 Face feature detection part 12 Feature selection / measurement part 13 Aperture ratio calculation part 14 Open determination part 15 Deletion detection part 16 Value update part 17 Value detection part 20 Face image 21 Mouth image 100 Steering 101 Instruments 102 Camera System

Claims (5)

Imaging means for acquiring a face image;
Feature point extracting means for extracting feature points from a mouth image included in the face image acquired by the imaging means;
An aspect ratio calculating means for calculating an aspect ratio of the mouth based on the feature points extracted by the feature point extracting means;
Mouth opening / closing state determination means for determining the opening / closing state of the mouth based on the aspect ratio calculated by the aspect ratio calculating means;
An opening time calculating means for calculating the opening time when the mouth open / closed state determining means is determined to be in the open state;
A non-expansion determining means for determining whether or not it is in a non-extended state based on the opening time calculated by the opening time calculating means;
The aspect ratio calculated by the aspect ratio calculation means is collected for a predetermined period, a histogram is created, the aspect ratio of the histogram is counted from the higher value, and the aspect ratio at a predetermined ratio of the entire histogram is determined to be open. Further comprising an open determination threshold value updating means for determining as a threshold value,
The open determination threshold value updating means compares the determined open determination threshold value with a predetermined range determined in advance by statistical data, and when the determined open determination threshold value is lower than the predetermined range. The minimum value of the predetermined range is determined as the open determination threshold value, and when the determined open determination threshold value is higher than the predetermined range, the maximum value of the predetermined range is determined as the open determination threshold value. yawn determination apparatus according to claim to Rukoto.   2. The non-extension determination apparatus according to claim 1, wherein the mouth open / close state determination unit determines that the mouth is in an open state when the aspect ratio is larger than a predetermined open determination threshold value.   2. The non-extension determination apparatus according to claim 1, wherein the non-extension determination unit determines that the open state is in an extended state when the opening time is longer than a predetermined period threshold value. On the basis of the opening determination threshold update means opening determination threshold updated by any of claims 1 to 3, characterized in that it comprises updating means for updating the predetermined time period threshold 2. The failure extension determination device according to item 1 . Calculate the aspect ratio of the mouth based on the feature points extracted from the mouth image included in the face image, compare the aspect ratio with a predetermined open determination threshold, determine the open / closed state of the mouth,
By comparing with a predetermined time period threshold has been opened time determining the port open, a yawn determination method of determining whether yawn state has occurred,
A histogram is created by collecting the aspect ratio calculated for a predetermined period, the aspect ratio of the histogram is counted from the upper value, and the aspect ratio at a predetermined ratio of the entire histogram is determined as an open determination threshold value,
The determined open determination threshold value is compared with a predetermined range determined by statistical data. If the determined open determination threshold value is lower than the predetermined range, the minimum value of the predetermined range is determined to be open. yawn determination method determines as the threshold, when the determined opening determination threshold is higher than said predetermined range, it characterized that you determine the maximum value of the predetermined range as an open determination threshold .

Images