JP2010217955A - Detection device, evaluation device and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more properly use a detection result of a state of an eye of a person. <P>SOLUTION: A face detection part 31 detects at least one of size, a shape, and a position of at least a part of a face of the person by use of an image captured by an imaging device 11. An eye state detection part 33 detects at least one of opening/closing and a shape of the eye of the person, and a view line direction by use of the image captured by the imaging device 11. A reliability evaluation part 22 evaluates reliability of the detection result by the eye state detection part 33 based on a change of a detection result by the face detection part 31. The detection device can be applied to a device detecting the state of the eye of a driver, for example. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a detection device, an evaluation device and method, and a program, and more particularly, to a detection device, an evaluation device and method, and a program that are suitable for detecting the state of a human eye.

  Conventionally, many devices and techniques for detecting the driver's line of sight and the open / closed state of the eyes have been proposed in order to detect the driver's looking aside or falling asleep.

  By the way, in order to accurately detect the driver's line of sight and eye open / closed state, it is necessary to accurately detect the position of the driver's face, the position of each part of the eye, the orientation of the face, and the like. . However, most conventional devices and technologies are constructed on the assumption that they can be detected correctly (see, for example, Patent Document 1).

  However, actually, the position of the driver's face, the position of each part of the eye, the orientation of the face, and the like are not correctly detected, and the detection result of the line of sight and the open / closed state of the eyes may become unstable. And if the detection result of the driver's line of sight or eye open / closed state becomes unstable, there is a risk that a malfunction will occur in the subsequent device, or a false detection such as a driver's side look or doze will occur. .

  Therefore, conventionally, the state where the eye is detected is not within a predetermined region in the image, the state where the distance between the detected eyes is not within the predetermined range, or the displacement of the detected vertical position of the eyes of both eyes is predetermined. When a state that is not within the range continues for a predetermined time, it has been proposed to turn on the buzzer and warn the driver that the eye cannot be recognized or that an error has been detected (see, for example, Patent Document 2).

JP 2006-109980 A JP-A-6-344801

  However, in the invention described in Patent Document 2, since the region and value range in the image used for determination are fixed, it is difficult to flexibly cope with individual differences such as face creation, physique, and driving posture. As a result, it is possible to overlook a state in which eye recognition is impossible or a false detection, and a malfunction or a false detection may occur in a subsequent device.

  The present invention has been made in view of such a situation, and makes it possible to more appropriately use the detection result of the human eye state.

  A detection device according to a first aspect of the present invention includes a face detection unit that detects at least one of the size, shape, and position of at least a part of a human face using an image captured by an imaging device; The eye state detecting means for detecting at least one of the opening / closing and shape of the human eye and the line-of-sight direction, and the change in the first detection result by the face detecting means And a reliability evaluation means for evaluating the reliability of the two detection results.

  In the detection device according to the first aspect of the present invention, at least one of the size, shape, and position of at least a part of a person's face is detected using an image captured by the imaging device, and the person is detected using the image. At least one of the eye opening / closing and shape and the line-of-sight direction is detected, and the reliability of the second detection result by the eye state detection means is evaluated based on the change in the first detection result by the face detection means. The

  Therefore, the reliability of the detection result of the human eye state can be accurately evaluated. As a result, for example, it becomes possible to use only the detection result with high reliability, and the detection result of the human eye state can be used more appropriately.

  The face detection means, eye state detection means, and reliability evaluation means are constituted by, for example, a CPU (Central Processing Unit).

  The reliability evaluation unit can evaluate the reliability of the second detection result based on the difference between the first detection result and the average value of the first detection result.

  Thereby, the reliability of the detection result of the human eye state can be more accurately evaluated.

  The detection apparatus further includes a face direction detection unit that detects the direction of the human face, and the reliability evaluation unit includes a change in the human face direction detected by the face direction detection unit within a predetermined range. The average value of the first detection results when the state is continuous for a predetermined period is obtained, and the second detection result is based on the difference between the first detection result and the obtained average value of the first detection results. Can be evaluated.

  Thereby, the reliability of the detection result of the human eye state can be evaluated in a flexible manner corresponding to individual differences and human state differences.

  The detection device may further include an output unit that outputs the second detection result and information indicating the reliability of the second detection result.

  Thereby, in the latter apparatus using the detection result of the human eye state, the detection result can be used more appropriately based on the reliability.

  This output means is constituted by, for example, a CPU (Central Processing Unit).

  When the first detection result includes a plurality of items, the reliability evaluation unit evaluates the reliability of the second detection result for each item of the first detection result, and the second detection result for each item. The reliability of the final second detection result is evaluated on the basis of the reliability evaluation result, and the output means further outputs the reliability evaluation result of the second detection result for each item. Can do.

  Thereby, it is possible to know the cause that is the basis for evaluating the reliability of the detection result in the subsequent apparatus that uses the detection result of the human eye state.

  The detection device outputs the second detection result when the reliability of the second detection result is equal to or higher than a predetermined threshold value, and outputs the second detection result when the reliability of the second detection result is lower than the predetermined threshold value. An output means that does not output the detection result of 2 can be further provided.

  This makes it possible to reliably use only the detection result with high reliability in the subsequent apparatus that uses the detection result of the human eye state.

  This output means is constituted by, for example, a CPU (Central Processing Unit).

  The face detection means can detect at least one of the size, shape and position of at least a part of the human face excluding the human eyes.

  Thereby, the reliability of the detection result of the human eye state can be evaluated based on the face portion of the person excluding the human eye.

  The face detection means can detect at least one of the size, shape and position of the entire human face.

  Thereby, the reliability of the detection result of the human eye state can be evaluated based on at least one of the size, shape, and position of the entire human face.

  An evaluation device according to a second aspect of the present invention includes a face detection unit that detects at least one of the size, shape, and position of at least a part of a human face using an image captured by an imaging device, and an image In the evaluation apparatus for evaluating the reliability of the detection result of the detection device comprising: eye state detection means for detecting at least one of the eye opening / closing and shape and the eye gaze direction using And reliability evaluation means for evaluating the reliability of the second detection result by the eye state detection means based on the change in the detection result of 1.

  In the evaluation apparatus according to the second aspect of the present invention, the reliability of the second detection result by the eye state detection unit is evaluated based on the change in the first detection result by the face detection unit.

  Therefore, the reliability of the detection result of the human eye state can be accurately evaluated. As a result, for example, it becomes possible to use only the detection result with high reliability, and the detection result of the human eye state can be used more appropriately.

  The face detection means, eye state detection means, and reliability evaluation means are constituted by, for example, a CPU (Central Processing Unit).

  The reliability evaluation unit can evaluate the reliability of the second detection result based on the difference between the first detection result and the average value of the first detection result.

  Thereby, the reliability of the detection result of the human eye state can be more accurately evaluated.

  The detection apparatus further includes a face direction detection unit that detects the direction of the human face, and the reliability evaluation unit includes a change in the human face direction detected by the face direction detection unit within a predetermined range. The average value of the first detection results when the state is continuous for a predetermined period is obtained, and the second detection result is based on the difference between the first detection result and the obtained average value of the first detection results. Can be evaluated.

  Thereby, the reliability of the detection result of the human eye state can be evaluated in a flexible manner corresponding to individual differences and human state differences.

  An evaluation method according to a second aspect of the present invention includes a face detection unit that detects at least one of the size, shape, and position of at least a part of a human face using an image captured by an imaging device, and an image An evaluation device that evaluates the reliability of a detection result of a detection device that includes an eye state detection unit that detects at least one of the opening / closing and shape of a human eye and the line-of-sight direction using a face detection unit. And a step of evaluating the reliability of the second detection result by the eye state detection means based on the change in the detection result of 1.

  A program according to a second aspect of the present invention includes a face detection unit that detects at least one of the size, shape, and position of at least a part of a human face using an image captured by an imaging device; A computer that evaluates the reliability of the detection result of the detection device including the eye state detection unit that detects at least one of the opening / closing and shape of the human eye and the line-of-sight direction is used. Based on the change in the detection result, a process including a step of evaluating the reliability of the second detection result by the eye state detection unit is executed.

  Therefore, the reliability of the detection result of the human eye state can be accurately evaluated. As a result, for example, it becomes possible to use only the detection result with high reliability, and the detection result of the human eye state can be used more appropriately.

  The detection device, face detection means, and eye state detection means are constituted by, for example, a CPU (Central Processing Unit).

  According to the first aspect or the second aspect of the present invention, the reliability of the detection result of the human eye state can be accurately evaluated. Moreover, according to the 1st side surface or 2nd side surface of this invention, the detection result of a human eye state can be utilized more appropriately.

It is a block diagram which shows one Embodiment of the eye state detection system to which this invention is applied. It is a figure for demonstrating the specific example of the item detected by a face detection part. It is a figure for demonstrating the outline | summary of the process of a reliability evaluation part. It is a flowchart for demonstrating an eye state detection process. It is a flowchart for demonstrating the detail of a reliability evaluation process. It is a flowchart for demonstrating the detail of a reliability evaluation process. It is a state transition diagram of a reliability evaluation part. It is a flowchart for demonstrating the detail of a face detection result evaluation process. It is a figure which shows the structural example of a computer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an embodiment of an eye state detection system to which the present invention is applied. The eye state detection system 1 in FIG. 1 is a system that is provided in a vehicle, for example, and detects the state of the eyes of a driver who drives the vehicle, and performs processing according to the detection result. The eye state detection system 1 is configured to include an imaging device 11, a detection device 12, and an application device 13. The detection device 12 is configured to include a detection unit 21, a reliability evaluation unit 22, and an output control unit 23. The detection unit 21 is configured to include a face detection unit 31, a face state detection unit 32, and an eye state detection unit 33.

  Hereinafter, an example in which the eye state detection system 1 is provided in a vehicle will be described. Hereinafter, a vehicle provided with the eye state detection system 1 is referred to as a host vehicle.

  The imaging device 11 images the face of the driver who drives the vehicle, and supplies an image (hereinafter referred to as a face image) obtained as a result to the face detection unit 31.

  Note that the imaging device 11 is desirably installed at a position where the face of the driver sitting in the driver's seat can be reliably imaged regardless of differences in the driver's physique and posture. For example, the imaging device 11 is installed on a steering column and images a driver from the front, or installed near the base of a center mirror (room mirror) in the own vehicle (where the center mirror is attached to the own vehicle). The driver is imaged from the upper left diagonal direction.

  The detection device 12 detects the eye state of the driver based on the face image. The detection device 12 evaluates the reliability of the detection result of the driver's eye state. The detection device 12 supplies the detection result of the driver's eye state and information indicating the reliability to the application device 13.

  More specifically, the face detection unit 31 of the detection unit 21 of the detection device 12 detects the driver's face in the face image. Then, the face detection unit 31 supplies the face state detection unit 32 with information indicating the face image and the detection result of the driver's face. Further, the face detection unit 31 supplies information indicating the detection result of the driver's face to the reliability evaluation unit 22.

  Hereinafter, as illustrated in FIG. 2, the face detection unit 31 performs a position X in the X-axis direction and a position Y in the Y-axis direction of the driver's face in the face image and a face in each frame of the face image. An example in the case where the size S is detected will be described.

  The face state detection unit 32 detects the state of the driver's face based on the face image and the detection result of the driver's face by the face detection unit 31. The face state detection unit 32 supplies information indicating the face image, the detection result of the driver's face, and the detection result of the driver's face state to the eye state detection unit 33. Further, the face state detection unit 32 supplies information indicating the detection result of the driver's face state to the reliability evaluation unit 22.

  Hereinafter, an example in which the face state detection unit 32 detects the driver's face orientation as the driver's face state will be described.

  The eye state detection unit 33 is based on the face image, the detection result of the driver's face by the face detection unit 31, and the detection result of the driver's face state by the face state detection unit 32. Is detected. The eye state detection unit 33 supplies information indicating the detection result of the driver's eye state to the output control unit 23.

  Hereinafter, an example in which the eye state detection unit 33 detects the driver's eye direction and the eye open / close state as the driver's eye state will be described.

  Based on the detection result of the driver's face by the face detection unit 31 and the detection result of the driver's face state by the face state detection unit 32, the reliability evaluation unit 22 performs the driver's face detection by the eye state detection unit 33. Evaluate the reliability of eye state detection results.

  As will be described in detail later, the reliability evaluation unit 22 monitors the driver's face position X and position Y and size S by the face detection unit 31 in time series, as shown in FIG. 3, for example. . Then, when at least one of the face position X, the face position Y, and the face size S changes discontinuously as in the part surrounded by the dotted line in the figure, the face detection unit 31 uses the eye state detection unit 33. Evaluate that the reliability of the detection result is low. The reliability evaluation unit 22 supplies the output control unit 23 with reliability information indicating the reliability of the detection result of the driver's eye state.

  The output control unit 23 controls the detection result of the driver's eye state and the output of the information indicating the reliability to the application device 13.

  The application device 13 is a device that uses the detection result of the driver's eye state. For example, the application device 13 detects a driver's look-aside or doze based on the detection result of the driver's eye state, and issues a warning to alert the driver.

  Next, the details of the processing of the eye state detection system 1 will be described with reference to FIGS.

  First, the eye state detection process executed by the eye state detection system 1 will be described with reference to the flowchart of FIG. This process is started, for example, when the power supply of the accessory (ACC) of the own vehicle is turned on, and is ended when the power supply of the accessory is turned off.

  In step S1, the eye state detection system 1 acquires a face image. Specifically, the imaging device 11 images the face of the driver of the own vehicle, and inputs a face image obtained as a result to the face detection unit 31.

  In step S2, the face detection unit 31 detects a face. Specifically, the face detection unit 31 detects the size and position of the driver's face in the face image using a predetermined method. Then, the face detection unit 31 supplies the face image and information indicating the detection result of the size and position of the driver's face to the face state detection unit 32. The face detection unit 31 supplies information indicating the detection result of the size and position of the driver's face to the reliability evaluation unit 22.

  Furthermore, the face detection unit 31 stores data indicating the size and position of the detected driver's face in a ring buffer (not shown). As a result, the detection result of the size and position of the driver's face is stored for a predetermined period, and old data is overwritten with new data.

  Note that the method by which the face detection unit 31 detects the size and position of the face is not limited to a specific method, and any method can be adopted.

  In step S3, the face state detection unit 32 detects the face state. Specifically, the face state detection unit 32 detects the orientation of the driver's face based on the face image and the detection result of the size and position of the driver's face using a predetermined method. The direction of the driver's face is represented, for example, by the angle of the driver's face relative to the front direction of the host vehicle, with the left direction being a negative value and the right direction being a positive value. Hereinafter, the angle of the driver's face direction detected by the face state detection unit 32 is α. Then, the face state detection unit 32 supplies information indicating the face image, the detection result of the size and position of the driver's face, and the detection result of the direction of the driver's face to the eye state detection unit 33. Further, the face state detection unit 32 supplies information indicating the detection result of the driver's face direction to the reliability evaluation unit 22.

  Further, the face state detection unit 32 stores data indicating the detected driver's face orientation in a ring buffer (not shown). As a result, the detection result of the driver's face orientation is stored for a predetermined period, and old data is overwritten with new data.

  Note that the method by which the face state detection unit 32 detects the orientation of the face is not limited to a specific method, and any method can be adopted.

  In step S4, the eye state detection unit 33 detects the eye state. Specifically, the face state detection unit 32 uses a predetermined method to drive based on the face image, the detection result of the driver's face size and position, and the detection result of the driver's face direction. The direction of the eyes of the person and the open / closed state of the eyes are detected. The eye state detection unit 33 supplies eye state detection information indicating the direction of the driver's line of sight and the detection result of the eye open / closed state to the output control unit 23.

  Note that the method by which the eye state detection unit 33 detects the direction of the line of sight and the open / closed state of the eyes is not limited to a specific method, and any method can be employed. Further, the eye state detection unit 33 does not use the detection result of the driver's face size and position by the face detection unit 31 and the detection result of the driver's face direction by the face state detection unit 32, and the driver The direction of the line of sight and the open / closed state of the eyes may be detected.

  In step S5, the reliability evaluation unit 22 performs a reliability evaluation process. Here, details of the reliability evaluation processing will be described with reference to the flowcharts of FIGS. 5 and 6.

  In step S51, the reliability evaluation unit 22 determines whether or not the current state is being initialized.

  Here, as shown in FIG. 7, the state of the reliability evaluation unit 22 transitions to three states during initial setting, during determination, and during state-of-appearance determination. At the start of the eye state detection process, the state of the reliability evaluation unit 22 is set during initial setting.

  If it is determined in step S51 that the current state is being initialized, the process proceeds to step S52.

  In step S52, the reliability evaluation unit 22 determines whether -th_α <α (the angle of the driver's face) <th_α. If it is determined that −th_α <α <th_α, that is, if the driver's left and right face orientation is within a predetermined range, the process proceeds to step S53.

  Note that th_α is a predetermined threshold value, for example, 15 degrees.

  In step S <b> 53, the reliability evaluation unit 22 determines whether the state where the face orientation is stable continues for Na frames based on data stored in a ring buffer (not shown). When it is determined that the face orientation is stable, that is, the state where -th_α <α <th_α is continued for Na frames (for example, 20 frames), the process proceeds to step S54.

  Na is the number of frames that determines the period for determining the stability of the driver's face orientation, and is set by the user, for example.

  In step S54, the reliability evaluation unit 22 obtains a weighted average of the size and position of the face during the Nb frame. Specifically, the reliability evaluation unit 22 reads the driver's face size and position data of the latest Nb frame (for example, 10 frames) from a ring buffer (not shown), and obtains a weighted average thereof.

  Nb is the number of frames that determines the calculation period of the weighted average of the face size and position, and is set by the user, for example. Further, a weighted average of the face size and position in a state where Nb ≦ Na is satisfied and the driver's face orientation is stable is obtained.

  In step S55, the reliability evaluation unit 22 changes the state of the reliability evaluation unit 22 during determination. Thereafter, the reliability evaluation process ends.

  On the other hand, if it is determined in step S53 that the face orientation is not stable during the Na frame, the processes in steps S54 and S55 are skipped, and the reliability evaluation process ends.

  On the other hand, if it is determined in step S52 that −th_α <α <th_α is not satisfied, that is, if the driver's left and right face orientation exceeds the predetermined range, the process proceeds to step S56.

  In step S56, the reliability evaluation unit 22 resets the average value of the face size and position. That is, the reliability evaluation unit 22 sets the average value of the face size and position to 0. Thereafter, the reliability evaluation process ends.

  On the other hand, if it is determined in step S51 that the current state is not being initialized, the process proceeds to step S57.

  In step S57, the reliability evaluation unit 22 determines whether the current state is being determined. If it is determined that the current state is being determined, the process proceeds to step S58.

  In step S58, the reliability evaluation unit 22 performs a face detection result evaluation process. Here, details of the face detection result evaluation process will be described with reference to the flowchart of FIG.

  In step S101, the reliability evaluation unit 22 determines whether or not | the average value of the face size−the detected value | ≦ th_s. Here, th_s is a predetermined threshold value. That is, the reliability evaluation unit 22 determines whether or not the driver's face size S detected this time is within a predetermined range from the average value. If it is determined that | average value of face size−detected value | ≦ th_s, the process proceeds to step S102.

  In step S102, the reliability evaluation unit 22 sets the value of the reliability evaluation flag (size) to 1. Thereafter, the process proceeds to step S104.

  On the other hand, if it is determined in step S101 that | average value of face size S−detected value |> th_s, the process proceeds to step S103.

  In step S103, the reliability evaluation unit 22 sets the value of the reliability evaluation flag (size) to 0. Thereafter, the process proceeds to step S104.

  In step S104, the reliability evaluation unit 22 determines whether or not | average value of face position X−detected value | ≦ th_x. Here, th_x is a predetermined threshold value. That is, the reliability evaluation unit 22 determines whether or not the position of the driver's face detected this time in the X-axis direction is within a predetermined range from the average value. If it is determined that | average value of face position X−detected value | ≦ th_x, the process proceeds to step S105.

  In step S105, the reliability evaluation unit 22 sets the value of the reliability evaluation flag (position X) to 1. Thereafter, the process proceeds to step S107.

  On the other hand, if it is determined in step S104 that | average value of face position X−detected value |> th_x, the process proceeds to step S106.

  In step S106, the reliability evaluation unit 22 sets the value of the reliability evaluation flag (position X) to 0. Thereafter, the process proceeds to step S107.

  In step S107, the reliability evaluation unit 22 determines whether or not | average value of face position Y−detected value | ≦ th_y. Here, th_y is a predetermined threshold value. That is, the reliability evaluation unit 22 determines whether or not the position of the driver's face in the Y-axis direction is within a predetermined range from the average value. If it is determined that | the average value of the face position Y−the detected value | ≦ th_y, the process proceeds to step S108.

  In step S108, the reliability evaluation unit 22 sets the value of the reliability evaluation flag (position Y) to 1. Thereafter, the process proceeds to step S110.

  On the other hand, if it is determined in step S107 that | average value of face position Y−detected value |> th_y, the process proceeds to step S109.

  In step S109, the reliability evaluation unit 22 sets the value of the reliability evaluation flag (position Y) to 0. Thereafter, the process proceeds to step S110.

  In step S110, the reliability evaluation unit 22 determines whether or not the reliability evaluation flag (size) = 1. If it is determined that the reliability evaluation flag (size) = 1, the process proceeds to step S111.

  In step S111, the reliability evaluation unit 22 sets the value of the reliability cause flag (size) to 1. Thereafter, the process proceeds to step S113.

  On the other hand, if it is determined in step S110 that the reliability evaluation flag (size) is not 1, the process proceeds to step S112.

  In step S112, the reliability evaluation unit 22 sets the value of the reliability cause flag (size) to 0. Thereafter, the process proceeds to step S113.

  In step S113, the reliability evaluation unit 22 determines whether or not the reliability evaluation flag (position X) = 1. If it is determined that the reliability evaluation flag (position X) = 1, the process proceeds to step S114.

  In step S114, the reliability evaluation unit 22 sets the value of the reliability cause flag (position X) to 1. Thereafter, the process proceeds to step S116.

  On the other hand, if it is determined in step S113 that the reliability evaluation flag (position X) is not 1, the process proceeds to step S115.

  In step S115, the reliability evaluation unit 22 sets the value of the reliability cause flag (position X) to 0. Thereafter, the process proceeds to step S116.

  In step S116, the reliability evaluation unit 22 determines whether or not the reliability evaluation flag (position Y) = 1. If it is determined that the reliability evaluation flag (position Y) = 1, the process proceeds to step S117.

  In step S117, the reliability evaluation unit 22 sets the value of the reliability cause flag (position Y) to 1. Thereafter, the process proceeds to step S119.

  On the other hand, if it is determined in step S116 that the reliability evaluation flag (position Y) is not 1, the process proceeds to step S118.

  In step S118, the reliability evaluation unit 22 sets the value of the reliability cause flag (position Y) to 0. Thereafter, the process proceeds to step S119.

  In step S119, the reliability evaluation unit 22 obtains the face rectangle reliability. Here, the face rectangle reliability is a value representing the reliability of a rectangle representing the area of the driver's face detected from the face image. The reliability evaluation unit 22 uses the logical product (AND) of the reliability cause determination flag (size), the reliability cause determination flag (position X), and the reliability cause determination flag (position Y) as the face rectangle reliability. Ask. That is, the face rectangle reliability is evaluated based on the size and position of the driver's face detected from the face image. When the face rectangle reliability is 1, the possibility that the driver's face is accurately detected by the face detection unit 31 is high, and the reliability of the detection result of the driver's line-of-sight direction and eye open / closed state is also high. It is estimated to be. On the other hand, when the face rectangle reliability is 0, the possibility that the driver's face is accurately detected by the face detection unit 31 is low, and the reliability of the detection result of the driver's line-of-sight direction and eye open / closed state is also low. It is estimated to be. Thereafter, the face detection result evaluation process ends.

  Thus, the reliability of the detection result of the line-of-sight direction and the open / closed state of the eyes for each item of the face size, position X and position Y of the driver detected by the face detection unit 31 by the face detection result evaluation process is The face rectangle reliability indicating the reliability of the final detection result is obtained based on the evaluation result.

  Returning to FIG. 6, in step S <b> 59, the reliability evaluation unit 22 outputs an evaluation result. That is, the reliability evaluation unit 22 obtains information including the values of the face rectangle reliability, the reliability cause determination flag (size), the reliability cause determination flag (position X), and the reliability cause determination flag (position Y). The eye state detection unit 33 outputs the information to the output control unit 23 as reliability information indicating the reliability of the driver's line-of-sight direction and the eye opening / closing state. In addition, when the state of itself is in the initial setting, the reliability evaluation unit 22 sets information indicating that the initial setting is still in progress and the reliability cannot be evaluated, and outputs the reliability information.

  In step S60, the reliability evaluation unit 22 determines whether face detection is successful based on the face rectangle reliability. If the value of the face rectangle reliability is 1, the reliability evaluation unit 22 determines that face detection is successful, and the process proceeds to step S63.

  In step S61, the reliability evaluation unit 22 updates the average value of the face size and position. Specifically, for example, the reliability evaluation unit 22 uses a ring (not shown) of the driver's face size and position data of the latest Nx frame (for example, 10 frames) out of frames in which face detection has succeeded. Read from the buffer and find its weighted average. And the reliability evaluation part 22 updates the average value used for a face detection evaluation process to the calculated | required average value. Thereafter, the reliability evaluation process ends.

  On the other hand, in step S60, if the value of the face rectangle reliability is 0, the reliability evaluation unit 22 determines that face detection has failed, and the process proceeds to step S62.

  In step S62, the reliability evaluation unit 22 determines whether face detection has failed for Ny frames in succession. If it is determined that face detection has failed continuously for Ny frames (for example, 10 frames), the process proceeds to step S63.

  Ny is the number of frames that defines a period for determining whether or not to change the state of the reliability evaluation unit 22 from during determination to during appearance determination, and is set by the user, for example.

  In step S <b> 63, the reliability evaluation unit 22 changes the state during the appearance determination. Thereafter, the reliability evaluation process ends.

  On the other hand, if it is determined in step S62 that face detection has not yet failed in Ny frames (for example, 10 frames), the process in step S63 is skipped, and the reliability evaluation process ends.

  On the other hand, when it is determined in step S57 that the current state is not being determined, that is, the current state is being determined to be in a state of watching, the process proceeds to step S64.

  In step S64, the face detection result evaluation process is executed in the same manner as in step S58.

  In step S65, as in the process of step S59, reliability information indicating the reliability of the driver's line-of-sight direction and eye open / closed state by the eye state detection unit 33 is output from the reliability evaluation unit 22 to the output control unit 23. Is done. However, the reliability evaluation unit 22 corrects the value of the face rectangle reliability to 0 and outputs the current state because the current state is being determined.

  In step S66, as in the process of step S60, it is determined whether face detection is successful. If it is determined that face detection is successful, the process proceeds to step S67.

  In step S67, the reliability evaluation unit 22 determines whether face detection has succeeded in Nj frames continuously. If it is determined that face detection has succeeded continuously for Nj frames (for example, 10 frames), the process proceeds to step S68.

  Nj is the number of frames that determines a period for determining whether or not to change the state of the reliability evaluation unit 22 from during the state determination to during the determination, and is set by the user, for example.

  In step S68, the reliability evaluation unit 22 changes the state of the reliability evaluation unit 22 during determination. Thereafter, the reliability evaluation process ends.

  On the other hand, if it is determined in step S67 that face detection has not yet succeeded in Nj frames, the process in step S68 is skipped, and the reliability evaluation process ends.

  On the other hand, if it is determined in step S66 that face detection has failed, the process proceeds to step S69.

  In step S69, the reliability evaluation unit 22 determines whether face detection has failed for Ni frames in succession. If it is determined that face detection has failed continuously for Ni frames (for example, 10 frames), the process proceeds to step S70.

  Note that Ni is the number of frames that defines a period for determining whether or not to change the state of the reliability evaluation unit 22 from the state determination to the initial setting, and is set by the user, for example.

  In step S <b> 70, the reliability evaluation unit 22 changes the state of the reliability evaluation unit 22 during initial setting. Thereafter, the reliability evaluation process ends.

  On the other hand, if it is determined in step S69 that the face detection has not yet failed for Ni frames, the process in step S70 is skipped, and the reliability evaluation process ends.

  Returning to FIG. 4, in step S <b> 6, the output control unit 23 outputs the detection result. That is, the output control unit 23 outputs the eye state detection information acquired from the eye state detection unit 33 and the reliability information acquired from the reliability evaluation unit 22 to the application device 13.

  At this time, the eye state detection information may be output only when the face rectangle reliability value is 1, and the eye state detection information may not be output when the face rectangle reliability value is 0. Further, when the eye state detection information is not output, the reliability information may not be output, or only the reliability information may be output.

  In step S7, the application device 13 uses the detection result. For example, the application device 13 detects the driver's look-aside based on the detection result of the driver's line of sight, or detects the driver's doze based on the detection result of the open / closed state of the driver's eyes. Or do. When the application device 13 determines that the driver is looking aside or asleep, the application device 13 displays an image on the display, outputs a warning sound or a warning message, or lights such as an LED (Light Emitting Diode). Warning is given to the driver by lighting or flashing, or by vibrating the steering wheel or driver's seat.

  Further, the application device 13 stops the driver's aside and dozing detection processing while the value of the face rectangle reliability included in the reliability information is zero. The application device 13 operates based on, for example, the reliability cause determination flag (size), the reliability cause determination flag (position X), and the reliability cause determination flag (position Y) included in the reliability information. The driver is notified of the cause of stoppage of the driver's aside and dozing detection by an image, sound, light, or the like.

  Thereafter, the process returns to step S1, and the processes of steps S1 to S7 are repeatedly executed.

  As described above, the reliability of the detection result of the driver's line-of-sight direction and the open / closed state of the eyes can be accurately evaluated in a flexible manner corresponding to the individual differences and the state differences of the drivers. As a result, the application device 13 can more appropriately use the detection result of the driver's line-of-sight direction and the eye open / closed state. That is, for example, in the application device 13, an inappropriate detection result with low reliability is used, and it is possible to prevent erroneous detection or malfunction. Further, since the cause of the decrease in reliability can be known, for example, the application device 13 can use the information to notify the driver of the cause or perform processing for removing the cause. Become.

  In the above description, an example in which reliability is evaluated based on the size and position of the entire driver's face has been shown. However, at least one of the size, shape, and position of the driver's face is detected. You may make it evaluate reliability based on a result. For example, if the position of parts other than the driver's eyes (for example, mouth, nose, eyebrows, etc.) deviates more than the average value, it is evaluated that the reliability is low, or the change in the shape of the nose, ears, etc. is small If the size or shape of the facial part deviates more than the average value, it can be considered that the reliability is low. Further, the reliability may be evaluated by combining the detection results of a plurality of face parts, or by combining the detection results of the entire face and the face parts.

  In the above description, an example is shown in which the reliability value is represented by binary values of 0 and 1. However, the reliability value may be represented by three or more types of values or may be represented by continuous values.

  Further, in the above description, an example of detecting the open / closed state of the eyes and the line-of-sight direction has been shown. However, the present invention is not limited to other eye states, for example, eye shape, eye color, pupil size, sunglasses The present invention can also be applied to the case where it is detected whether or not the eyes are hidden.

  Moreover, although the example which evaluates the reliability of the detection result of a driver | operator's eye state was shown in the above description, this invention is not limited to a driver | operator in particular, The other person's eye state is detected. It can also be used in some cases.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 9 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are connected to each other by a bus 304.

  An input / output interface 305 is further connected to the bus 304. The input / output interface 305 includes an input unit 306 including a keyboard, a mouse, and a microphone, an output unit 307 including a display and a speaker, a storage unit 308 including a hard disk and a non-volatile memory, and a communication unit 309 including a network interface. A drive 310 that drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 301 loads the program stored in the storage unit 308 to the RAM 303 via the input / output interface 305 and the bus 304 and executes the program, for example. Is performed.

  The program executed by the computer (CPU 301) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. It is recorded on a removable medium 311 which is a package medium composed of a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 308 via the input / output interface 305 by attaching the removable medium 311 to the drive 310. Further, the program can be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. In addition, the program can be installed in advance in the ROM 302 or the storage unit 308.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in the present specification, the term “system” means an overall apparatus composed of a plurality of apparatuses and means.

  Furthermore, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 Eye state detection system 11 Imaging device 12 Detection device 13 Application apparatus 21 Detection part 22 Reliability evaluation part 23 Output control part 31 Face detection part 32 Face state detection part 33 Eye state detection part

Claims (13)

Face detection means for detecting at least one of the size, shape, and position of at least a part of the human face using an image captured by an imaging device;
Eye state detecting means for detecting at least one of the opening and closing and shape of the human eye and the line-of-sight direction using the image;
And a reliability evaluation unit that evaluates the reliability of the second detection result by the eye state detection unit based on a change in the first detection result by the face detection unit. The detection apparatus according to claim 1, wherein the reliability evaluation unit evaluates reliability of the second detection result based on a difference between the first detection result and an average value of the first detection result. . Further comprising face orientation detection means for detecting the orientation of the person's face,
The reliability evaluation unit calculates an average value of the first detection results when a state in which the change in the face direction of the person detected by the face direction detection unit is within a predetermined range continues for a predetermined period. The detection device according to claim 2, wherein the reliability of the second detection result is evaluated based on a difference between the obtained first detection result and the average value of the obtained first detection results. The detection apparatus according to claim 1, further comprising output means for outputting the second detection result and information indicating reliability of the second detection result. When the first detection result is composed of a plurality of items,
The reliability evaluation means evaluates the reliability of the second detection result for each item of the first detection result, and based on the evaluation result of the reliability of the second detection result for each item, Evaluating the reliability of the final second detection result;
The detection device according to claim 4, wherein the output unit further outputs a reliability evaluation result of the second detection result for each item. When the reliability of the second detection result is greater than or equal to a predetermined threshold, the second detection result is output, and when the reliability of the second detection result is less than the predetermined threshold, the second The detection apparatus according to claim 1, further comprising an output unit that does not output the detection result. The detection device according to claim 1, wherein the face detection unit detects at least one of a size, a shape, and a position of at least a part of the human face excluding the human eye. The detection device according to claim 1, wherein the face detection unit detects at least one of a size, a shape, and a position of the entire face of the person. Face detection means for detecting at least one of the size, shape and position of at least a part of the person's face using an image picked up by an image pickup device; and opening and closing of the person's eyes using the image; In the evaluation apparatus for evaluating the reliability of the detection result of the detection apparatus including the shape and the eye state detection means for detecting at least one of the line-of-sight directions,
An evaluation apparatus comprising: a reliability evaluation unit that evaluates the reliability of the second detection result by the eye state detection unit based on a change in the first detection result by the face detection unit. The evaluation apparatus according to claim 9, wherein the reliability evaluation unit evaluates reliability of the second detection result based on a difference between the first detection result and an average value of the first detection result. . The detection device further includes a face orientation detection unit that detects a face orientation of the person,
The reliability evaluation unit calculates an average value of the first detection results when a state in which the change in the face direction of the person detected by the face direction detection unit is within a predetermined range continues for a predetermined period. The evaluation apparatus according to claim 10, wherein the reliability of the second detection result is evaluated based on the difference between the first detection result and the average value of the calculated first detection result. Face detection means for detecting at least one of the size, shape and position of at least a part of the person's face using an image picked up by an image pickup device; and opening and closing of the person's eyes using the image; An evaluation device that evaluates the reliability of the detection result of the detection device including a shape and an eye state detection unit that detects at least one of the line-of-sight directions,
An evaluation method comprising a step of evaluating the reliability of the second detection result by the eye state detection means based on a change in the first detection result by the face detection means. Face detection means for detecting at least one of the size, shape and position of at least a part of the person's face using an image picked up by an image pickup device; and opening and closing of the person's eyes using the image A computer that evaluates the reliability of the detection result of the detection device including the shape and the eye state detection unit that detects at least one of the line-of-sight directions,
A program for executing a process including a step of evaluating the reliability of the second detection result by the eye state detection means based on a change in the first detection result by the face detection means.

Images