JP4915413B2 - Detection apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To quickly and surely detect opening and closing of human eyes. <P>SOLUTION: In a detection apparatus, an opening and closing detection unit 121 detects the opening and closing degree, showing the degree of opening of a driver, on the basis of the face image of the driver captured by an imaging device 111. A viewing direction detecting unit 122 detects the viewing direction of a person, on the basis of the face image of the driver. A threshold-setting unit 123 sets a determination threshold for use in determination of opening and closing of eyes on the basis of the detected viewing direction of the driver. An opening and closing determining unit 124 determines the opening and the closing of the driver's eyes, in comparison of the detected opening and closing degree of the driver's eyes with the determination threshold set. The detection apparatus is applicable to, for example, on-vehicle opening and closing detection apparatuses for driver's eyes. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to a detection device and method, and a program, and more particularly, to a detection device and method for detecting opening and closing of a human eye, and a program.

  In recent years, driving that detects a driver's eye opening / closing degree from an image of a driver driving the vehicle, determines the opening / closing of the eye based on the detected opening / closing degree, and detects a drowsy driving or the like Development of support devices is thriving.

  However, the opening of a person's eyes changes depending on the line-of-sight direction in addition to the eye opening / closing operation. For example, images 1 to 4 in FIG. 1 are images of facial images taken when only the line of sight is moved up and down with the person 11 facing the fixed camera and when the eyes are closed. An example is schematically shown. The image 1 is an image when the line of sight of the person 11 faces 20 degrees upward, the image 2 is an image when the line of sight of the person 11 faces the front, and the image 3 is the person 11 The image 4 is an image when the person 11 is closed, and the image 4 is an image when the person 11 is closed. The distance between the upper and lower eyelids of the person 11 is, for example, 12 pixels in the image 1, 9 pixels in the image 2, 5 pixels in the image 3, and 3 pixels in the image 4.

  Therefore, depending on the setting of the threshold value, for example, as shown in the image 3, the person 11 may face down and the degree of opening / closing of the eyes is small, but it may be erroneously determined that the eyes are closed. When this is applied to the driving support device, for example, when the line of sight is dropped to look at a speedometer or the like, it may be erroneously determined that the eyes are closed.

  Therefore, conventionally, in addition to the opening / closing determination threshold for determining the opening / closing of the eyes, a anti-opening determination threshold is set, and the degree of opening / closing of the eyes is closer to the opening / closing determination threshold and more open than the opening / closing determination threshold. If the condition lasts for a certain period of time, if there is a feature such as little change in the degree of opening and closing of the eyes during that period, it is determined that it is a puffy eye, and if it is determined that it is not a puffy eye, it is proposed to make a closed eye determination (For example, refer to Patent Document 1).

  In addition, when the degree of opening / closing of the eyes is less than a predetermined threshold, the driving state and driving operation state of the vehicle are detected, and whether the eyes are closed due to falling asleep based on the driving state and driving operation state of the vehicle It has been proposed to determine whether the person is looking aside (for example, see Patent Document 2).

JP-A-11-66304 JP 2003-233803 A

  However, in the invention described in Patent Document 1, a predetermined time is required until it is determined whether the eye is closed or the eye is closed, and the determination is delayed.

  Moreover, in the invention described in Patent Document 2, it is necessary to obtain information indicating the driving operation state and the vehicle traveling state from the outside, and the setting of the conditions for accurately distinguishing the downward looking aside from the dozing is performed. Difficult and prone to misjudgment.

  The present invention has been made in view of such a situation, and makes it possible to detect opening and closing of a human eye quickly and accurately.

  According to another aspect of the present invention, there is provided an open / closed degree detecting means for detecting an open / closed degree indicating an opening degree of the person's eyes based on an image obtained by imaging a person's face, and based on the image, Gaze direction detecting means for detecting a gaze direction, threshold setting means for setting a judgment threshold used for judging eye opening / closing based on the detected gaze direction of the person, and the degree of opening / closing as the judgment threshold Open / close determining means for determining whether the person's eyes are opened or closed by comparison.

  In the detection device according to one aspect of the present invention, an open / closed degree indicating the degree of opening of the person's eyes is detected based on an image obtained by imaging a person's face, and the direction of the line of sight of the person is determined based on the image. Based on the detected direction of the person's line of sight, a determination threshold value used for determination of opening / closing of eyes is set, and the opening / closing of the person's eyes is determined by comparing the degree of opening / closing with the determination threshold value. Is done.

  Therefore, it is possible to detect opening and closing of human eyes quickly and accurately.

  The degree-of-opening / closing degree detecting means, the line-of-sight direction detecting means, the threshold setting means, and the opening / closing determining means are constituted by arithmetic devices such as a CPU (Central Processing Unit) and an ECU (Electronic Control Unit).

  The threshold setting means obtains the correction amount with respect to the detected direction of the person's line of sight based on predetermined data indicating the relationship between the direction of the line of sight and the correction amount for correcting the determination threshold. The determination threshold value can be set by correcting a preset value using the correction amount.

  Thereby, it is possible to detect opening and closing of a human eye quickly and accurately with a small amount of data.

  The threshold setting means can set the determination threshold for the detected direction of the person's line of sight based on predetermined data indicating the relationship between the direction of the line of sight and the determination threshold.

  Thereby, it is possible to detect opening and closing of human eyes quickly and accurately with a small amount of calculation.

  The threshold setting means obtains the amount of change in the direction of the person's line of sight based on the detection result by the line-of-sight direction detection means, and when the amount of change is within a predetermined range, The determination threshold value can be set based on a direction, and when the amount of change exceeds the range, the determination threshold value can be set to a value when the person's line of sight faces a predetermined direction.

  Thereby, erroneous detection of opening and closing of human eyes can be suppressed, and detection accuracy can be improved.

  The detection apparatus may further include a pupil detection unit that detects the human pupil in the image and a pupil reliability calculation unit that calculates a reliability of the detection result of the human pupil. In the means, when the reliability exceeds a predetermined threshold, the determination threshold is set based on the detected direction of the line of sight of the person, and when the reliability is equal to or lower than the predetermined threshold, the person The determination threshold value can be set to a value when the line of sight is directed in a predetermined direction.

  Thereby, erroneous detection of opening and closing of human eyes can be suppressed, and detection accuracy can be improved.

  The eye opening / closing detection method according to one aspect of the present invention or the processing executed by the computer according to the program according to one aspect of the present invention is based on an image obtained by capturing an image of the person's face. Detecting a degree, detecting a direction of the line of sight of the person based on the image, setting a determination threshold used for determining opening / closing of the eye based on the detected direction of the line of sight of the person, and Comparing the eyes with the determination threshold value to determine whether the human eye is opened or closed.

  In the eye opening / closing detection method or program according to one aspect of the present invention, an opening / closing degree indicating the degree of opening of the person's eyes is detected based on an image obtained by imaging a person's face, and the person's eyes are opened based on the image. The direction of the line of sight is detected, and based on the detected direction of the line of sight of the person, a determination threshold value used for determination of opening / closing of the eye is set, and by comparing the degree of opening / closing with the determination threshold value, Whether to open or close is determined.

  Therefore, it is possible to detect opening and closing of human eyes quickly and accurately.

  This step is executed by, for example, a CPU (Central Processing Unit), an ECU (Electronic control unit), or the like.

  As described above, according to one aspect of the present invention, it is possible to detect opening and closing of a human eye. In particular, according to one aspect of the present invention, it is possible to detect opening and closing of a human eye quickly and accurately.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, a first embodiment of the present invention will be described with reference to FIG. 2 and FIG.

  FIG. 2 is a block diagram showing a first embodiment of an eye opening / closing detection system to which the present invention is applied. The eye opening / closing detection system 101 of FIG. 2 is configured to include an imaging device 111 and an eye opening / closing detection device 112. The eye opening / closing detection device 112 is configured to include an opening / closing degree detection unit 121, a line-of-sight detection unit 122, a threshold setting unit 123, an opening / closing determination unit 124, and a result output unit 125.

  The imaging device 111 captures the face of the driver of a vehicle (hereinafter referred to as the own vehicle) provided with the eye opening / closing detection system 101, and an image obtained as a result (hereinafter referred to as a face image) This is supplied to the open / closed degree detector 121 and the line-of-sight direction detector 122 of the open / close detector 112.

  Note that the imaging device 111 is desirably installed at a position where the face of the driver who is driving while sitting in the driver's seat can be reliably imaged regardless of differences in the physique and posture of the driver. For example, the imaging device 111 is installed on the steering column so as to image the driver from the front, or the center in the vehicle so as to image the driver from the front left diagonally upward direction (or diagonally right upward). It is installed near the base of the mirror (room mirror) (around the center mirror is attached to the vehicle).

  The open / closed degree detection unit 121 detects an open / closed degree indicating the degree of opening of the driver's eyes based on the face image. The degree of opening / closing becomes larger as the driver's eyes open in the face image becomes larger, and becomes smaller as the eyes open smaller. The opening / closing degree detection unit 121 supplies information indicating the detected degree of opening / closing of the driver's eyes to the opening / closing determination unit 124.

  The gaze direction detection unit 122 detects the direction of the driver's gaze based on the face image. The gaze direction detection unit 122 supplies information indicating the detected driver's gaze direction to the threshold setting unit 123.

  As will be described later with reference to FIG. 3, the threshold setting unit 123 sets a determination threshold used to determine whether the driver's eyes are open or closed based on the driver's line-of-sight direction. The threshold setting unit 123 supplies information indicating the set determination threshold to the open / close determination unit 124.

  The open / close determining unit 124 determines whether the driver's eyes are open or closed by comparing the open / closed degree of the driver's eyes detected by the open / closed degree detecting unit 121 with a determination threshold set by the threshold setting unit 123. The open / close determination unit 124 supplies information indicating the determination result to the result output unit 125.

  The result output unit 125 outputs the determination result of the driver's eye opening / closing to a subsequent device that uses the determination result of the driver's eye opening / closing, such as a snooze detection device.

  Next, the eye opening / closing detection process executed by the eye opening / closing detection system 101 will be described with reference to the flowchart of FIG. This process is started, for example, when the engine of the host vehicle is started, and is ended when the engine is stopped.

  In step S1, the eye opening / closing detection system 101 acquires an image of the driver. That is, the imaging device 111 captures an image of the driver's face and supplies an image (face image) obtained as a result to the open / closed degree detection unit 121 and the line-of-sight direction detection unit 122.

  In step S2, the opening / closing degree detection unit 121 detects the opening / closing degree of the eyes. That is, the open / closed degree detection unit 121 detects the open / closed degree of the eyes of the driver shown in the face image using a predetermined method. The opening / closing degree detection unit 121 supplies information indicating the detected degree of opening / closing of the driver's eyes to the opening / closing determination unit 124.

  Note that the method by which the open / closed degree detection unit 121 detects the open / closed degree of the eyes is not limited to a specific method, and it is desirable to apply a method that can be detected more simply, quickly, and accurately. For example, methods disclosed in Japanese Patent No. 3143819, Japanese Patent Laid-Open No. 2000-123188, and the like can be applied.

  In step S3, the gaze direction detection unit 122 detects the gaze direction. That is, the line-of-sight direction detection unit 122 detects the direction of the driver's line of sight in the face image using a predetermined method. The gaze direction detection unit 122 supplies information indicating the detected driver's gaze direction to the threshold setting unit 123.

  Note that the method of detecting the gaze direction by the gaze direction detection unit 122 is not limited to a specific method, and it is desirable to apply a method that can detect the gaze direction more easily, quickly, and accurately. For example, it is possible to apply the methods disclosed in JP 2006-141862 A, JP 2003-271932 A, and the like.

  In step S4, the threshold value setting unit 123 sets a determination threshold value. Specifically, the threshold setting unit 123 sets a determination threshold TH corresponding to the current driver's line-of-sight direction based on an eye opening / closing determination threshold table created in advance. The threshold setting unit 123 supplies information indicating the set determination threshold TH to the open / close determination unit 124.

  Here, the eye opening / closing determination threshold value table is a table including data indicating the relationship between the driver's line-of-sight direction and the determination threshold value. For example, when the driver ’s line-of-sight direction is facing the front direction of the vehicle (the front direction of the vehicle), the line-of-sight direction is −10 degrees (vertical and horizontal directions are 0 degrees) The value of the determination threshold TH in the range of 10 degrees below to 0 degrees is x, and the value of the determination threshold TH in the range of 0 degrees to +10 degrees (up to 10 degrees) is y. In the eye opening / closing determination threshold value table, the line-of-sight direction is divided into several ranges, and a determination threshold value TH is set for each range.

  Moreover, the eye opening / closing determination threshold value table is created, for example, by the following procedure. First, change the line-of-sight direction of a driver who is driving while sitting in the driver's seat to the driver's eyes while changing the direction of the vehicle up, down, left and right, and diagonally, based on the case of facing the front of the vehicle. Imaging is performed by the imaging device 111. Next, based on the plurality of images obtained as a result, data indicating the relationship between the driver's line-of-sight direction and the degree of eye opening / closing is acquired. Based on the acquired data, the determination threshold TH for each gaze direction of the driver is determined, and the eye opening / closing determination threshold table is summarized in a table.

  In order to further improve the detection accuracy of the driver's eye opening / closing, it is desirable to create an eye opening / closing determination threshold value table for each individual and to use the table appropriately for each driver. However, although there is a possibility that the detection accuracy is slightly lowered, one eye opening / closing determination threshold value table may be created from data of a plurality of drivers, and the same table may be used for a plurality of drivers.

  In step S <b> 5, the open / close determination unit 124 determines whether or not the degree of eye open / close> the determination threshold. That is, the open / close determination unit 124 determines whether or not the open / closed degree of the driver's eyes detected by the open / closed degree detection unit 121 is greater than the determination threshold TH set by the threshold setting unit 123. If it is determined that the degree of opening / closing of the eyes> the determination threshold, the process proceeds to step S6.

  In step S6, the open / close determination unit 124 determines that the eyes are open. Thereafter, the process proceeds to step S8.

  On the other hand, if it is determined in step S5 that the degree of opening / closing of the eyes ≦ the determination threshold, the process proceeds to step S7.

  In step S7, the open / close determination unit 124 determines that the eyes are closed. Thereafter, the process proceeds to step S8.

  In step S8, the result output unit 125 outputs the determination result to the subsequent apparatus. Thereafter, the process returns to step S1, and the processes after step S1 are executed.

  In this way, regardless of the direction of the driver's line of sight, it is possible to detect the opening and closing of the driver's eyes simply, quickly and accurately without performing complicated processing.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In the second embodiment of the present invention, first, an initial threshold value that is an initial value of the determination threshold value is set, and the determination threshold value is set by correcting the initial threshold value based on the driver's line-of-sight direction. It is a thing.

  FIG. 4 is a block diagram showing a second embodiment of an eye opening / closing detection system to which the present invention is applied. In the figure, portions corresponding to those in FIG. 2 are denoted by the same reference numerals in the last two digits, and description of portions having the same processing will be omitted because it will be repeated.

  The eye opening / closing detection system 201 in FIG. 4 is configured to include an imaging device 211 and an eye opening / closing detection device 212. The eye open / close detection device 212 is configured to include an open / close degree detection unit 221, a gaze direction detection unit 222, an open / close determination unit 224, a result output unit 225, an initial threshold setting unit 231, and a threshold correction unit 232. .

  The opening / closing degree detection unit 221 supplies information indicating the detected degree of opening / closing of the eyes of the driver to the initial threshold setting unit 231 in addition to the opening / closing determination unit 224.

  The line-of-sight direction detection unit 222 supplies information indicating the detected driver's line-of-sight direction to the initial threshold value setting unit 231 and the threshold value correction unit 232.

  As will be described later with reference to FIG. 5, the initial threshold setting unit 231 sets an initial threshold that is an initial value of the determination threshold based on the detected degree of eye opening / closing of the driver and the line-of-sight direction. The initial threshold setting unit 231 supplies information indicating the set initial threshold to the threshold correction unit 232.

  As described later with reference to FIG. 5, the threshold correction unit 232 sets the determination threshold by correcting the initial threshold based on the detected driver's line-of-sight direction. The threshold correction unit 232 supplies information indicating the set determination threshold to the open / close determination unit 224.

  Next, the eye opening / closing detection process executed by the eye opening / closing detection system 201 will be described with reference to the flowchart of FIG. This process is started, for example, when the engine of the host vehicle is started, and is ended when the engine is stopped.

  In step S51, the initial threshold setting unit 231 sets an initial threshold. For example, the initial threshold value setting unit 231 acquires the detection result of the degree of opening / closing of the driver's eyes from the opening / closing degree detection unit 221 for a predetermined period after the eye opening / closing detection process is started. Then, the initial threshold value setting unit 231 calculates a statistical value (for example, average value, variance, etc.) of the acquired eye open / closed degree, and sets the initial threshold value THO based on the calculated statistical value. At this time, the detection result of the driver's gaze direction is acquired from the gaze direction detection unit 222, and the weight of the degree of opening and closing the eyes is set based on the gaze direction (for example, the weight is increased as the gaze direction is closer to the front direction of the vehicle). The statistical value used for setting the initial threshold value may be calculated. The initial threshold setting unit 231 supplies information indicating the set initial threshold THO to the threshold correction unit 232.

  Note that the method by which the initial threshold setting unit 231 sets the initial threshold is not limited to a specific method, and it is desirable to apply a method that can be set more simply, quickly, and accurately. For example, it is possible to apply the methods disclosed in Japanese Patent Application Laid-Open Nos. 2006-109980 and 2004-41485.

  Alternatively, an initial threshold may be set for each driver in advance, and the initial threshold setting unit 231 may authenticate the driver and select and set the initial threshold according to the driver. Alternatively, the driver selects and sets an initial threshold value corresponding to the driver by an operation such as the driver selecting and pressing the push button corresponding to the driver from among the push buttons set according to the driver. You may do it.

  The processing in steps S52 to S54 is the same as the processing in steps S1 to S3 in FIG. 3, and description thereof will be omitted because it will be repeated. Through these processes, the degree of opening and closing of the driver's eyes and the line-of-sight direction are detected based on the driver's face image.

  In step S55, the threshold correction unit 232 corrects the determination threshold. Specifically, the threshold correction unit 232 calculates a correction amount C corresponding to the current driver's line-of-sight direction based on a correction table created in advance. The threshold correction unit 232 calculates a determination threshold TH based on the following equation (1), and supplies information indicating the calculated determination threshold TH to the open / close determination unit 224.

  TH = THO × C (1)

  Here, the correction table is a table including data indicating the relationship between the driver's line-of-sight direction and the correction amount C. For example, when the driver's line-of-sight direction is facing the front direction of the vehicle, the line-of-sight direction is -10 degrees (10 degrees downward) to 0 in the vertical direction. In the correction table, the value of the correction amount C is x, and when the correction amount C is in the range of 0 to +10 degrees (up to 10 degrees), the correction amount C is y. It is divided into several ranges, and a correction amount C is set for each range.

  The correction table is created by the following procedure, for example. First, change the line-of-sight direction of a driver who is driving while sitting in the driver's seat to the driver's eyes while changing the direction of the vehicle up, down, left and right, and diagonally, based on the case of facing the front of the vehicle. Imaging is performed by the imaging device 211. And based on the several image obtained as a result, the data which show the relationship between a driver | operator's eyes | visual_axis direction and the opening / closing degree of eyes are acquired. Data indicating the relationship between the line-of-sight direction and the degree of eye opening / closing is acquired for a sufficient number of drivers. Based on the acquired data, statistics on the amount of change in the degree of eye opening / closing when the eyes are facing each direction other than the front with respect to the degree of eye opening / closing when the line of sight is facing the front of the vehicle are collected. Based on the result, the correction amount C for each gaze direction is determined, and the correction table is summarized in a table.

  The correction table may be created for each vehicle, for example, or may be created and provided for each vehicle type by an automobile manufacturer or the like.

  The processing in steps S56 to S59 is the same as the processing in steps S5 to S8 in FIG. 3, and the description thereof will be omitted because it will be repeated. Note that the opening / closing of the eyes of the driver is determined by these processes. Thereafter, the process returns to step S52, and the processes after step S52 are executed.

  In this way, regardless of the direction of the driver's line of sight, it is possible to detect the opening and closing of the driver's eyes simply, quickly and accurately without performing complicated processing. In addition, one type of correction table may be used regardless of the driver, and processing can be performed with a smaller amount of data.

  Next, a third embodiment of the present invention will be described with reference to FIGS. In the third embodiment of the present invention, the determination threshold is adjusted based on the amount of change in the driver's line-of-sight direction.

  FIG. 6 is a block diagram showing a third embodiment of an eye opening / closing detection system to which the present invention is applied. In the figure, portions corresponding to those in FIG. 2 are denoted by the same reference numerals in the last two digits, and description of portions having the same processing will be omitted because it will be repeated.

  The eye opening / closing detection system 301 of FIG. 6 is configured to include an imaging device 311 and an eye opening / closing detection device 312. The eye opening / closing detection device 312 includes an opening / closing degree detection unit 321, a gaze direction detection unit 322, a threshold setting unit 323, an open / close determination unit 324, a result output unit 325, and a gaze direction holding unit 331. .

  The gaze direction detection unit 322 supplies information indicating the detected gaze direction of the driver to the gaze direction holding unit 331 in addition to the threshold setting unit 323.

  As will be described later with reference to FIG. 7, the threshold setting unit 323 detects the driver's line-of-sight direction detected last time held in the line-of-sight direction holding unit 331 and the current detection detected by the line-of-sight detection unit 322. A determination threshold value is set based on the driver's line-of-sight direction. The threshold setting unit 323 supplies information indicating the set threshold to the open / close determination unit 324.

  Next, the eye opening / closing detection process executed by the eye opening / closing detection system 301 will be described with reference to the flowchart of FIG. This process is started, for example, when the engine of the host vehicle is started, and is ended when the engine is stopped.

  The processing in steps S101 and S102 is the same as the processing in steps S1 and S2 in FIG. 3, and the description thereof will be omitted because it will be repeated. By these processes, the degree of opening and closing of the driver's eyes is detected based on the driver's face image.

  In step S103, the line-of-sight direction detection unit 322 detects the line-of-sight direction of the driver in the same manner as the process performed by the line-of-sight direction detection unit 122 in step S3 of FIG. The gaze direction detection unit 322 supplies information indicating the detected driver's gaze direction to the threshold setting unit 323 and the gaze direction holding unit 331.

  In step S <b> 104, the gaze direction holding unit 331 holds information indicating the driver's gaze direction supplied from the gaze direction detection unit 322.

  In step S105, the threshold setting unit 323 determines whether or not the amount of change in the line-of-sight direction is within a predetermined range. Specifically, the threshold value setting unit 323 acquires information indicating the driver's line-of-sight direction detected in the previous processing of step S103 from the line-of-sight direction holding unit 331. The threshold setting unit 323 obtains a change amount between the previous driver's gaze direction acquired from the gaze direction holding unit 331 and the current driver's gaze direction acquired from the gaze direction detection unit 322. If the threshold setting unit 323 determines that the obtained change amount is within the predetermined range, the process proceeds to step S106. In addition, when the detection result of the last gaze direction is not hold | maintained, for example, a process progresses unconditionally to step S106.

  In step S <b> 106, the threshold setting unit 323 sets a determination threshold based on the current driver's line-of-sight direction, similar to the processing of the threshold setting unit 123 in step S <b> 4 of FIG. 3. Note that the process of step S106 is a process performed by determining that the reliability of the detection result of the driver's gaze direction this time is high when the change amount of the driver's gaze direction is within a predetermined range. . Thereafter, the process proceeds to step S108.

  On the other hand, if it is determined in step S105 that the amount of change in the line-of-sight direction exceeds a predetermined range, the process proceeds to step S107.

  In step S107, the threshold setting unit 323 sets the determination threshold to a value when the driver's line of sight is facing the front, based on the eye opening / closing determination threshold table. That is, regardless of the detection result of the driver's line-of-sight direction, the determination threshold is set to the same value. Note that the processing in step S107 is performed when the amount of change in the driver's line of sight exceeds a predetermined range, for example, due to factors such as the driver's eyes being closed. This process is performed by determining that the reliability of the detection result is low. Thereafter, the process proceeds to step S108.

  The processing in steps S108 to S111 is the same as the processing in steps S5 to S8 in FIG. Note that the opening / closing of the eyes of the driver is determined by these processes. Thereafter, the process returns to step S101, and the processes after step S101 are executed.

  In this way, when the reliability of the detection result of the driver's line-of-sight direction is low, erroneous detection of the opening / closing of the driver's eyes is suppressed, and the detection accuracy is improved.

  Note that, in the third embodiment of the present invention, the method for obtaining the amount of change in the driver's line-of-sight direction is not limited to the method described above.

  Next, a fourth embodiment of the present invention will be described with reference to FIGS. In the fourth embodiment of the present invention, the driver's pupil is detected, and the determination threshold is adjusted based on the reliability of the pupil detection result.

  FIG. 8 is a block diagram showing a fourth embodiment of an eye opening / closing detection system to which the present invention is applied. In the figure, portions corresponding to those in FIG. 2 are denoted by the same reference numerals in the last two digits, and description of portions having the same processing will be omitted because it will be repeated.

  The eye opening / closing detection system 401 of FIG. 8 is configured to include an imaging device 411 and an eye opening / closing detection device 412. The eye opening / closing detection device 412 includes an opening / closing degree detection unit 421, a gaze direction detection unit 422, a threshold setting unit 423, an opening / closing determination unit 424, a result output unit 425, a pupil detection unit 431, and a pupil reliability calculation unit 432. Configured to include.

  The imaging device 411 supplies the driver's face image to the pupil detection unit 431 in addition to the open / closed degree detection unit 421 and the gaze direction detection unit 422 of the eye open / close detection device 412.

  The pupil detection unit 431 detects the driver's pupil in the face image. The pupil detection unit 431 supplies information indicating the detection result of the face image and the driver's pupil to the pupil reliability calculation unit 432.

  The pupil reliability calculation unit 432 calculates the reliability of the pupil detection result by the pupil detection unit 431. The pupil reliability calculation unit 432 supplies information indicating the calculated reliability to the threshold setting unit 423.

  As described later with reference to FIG. 9, the threshold setting unit 423 sets a determination threshold based on the detected driver's gaze direction and the reliability of the calculated driver's pupil detection result. . The threshold setting unit 423 supplies information indicating the set threshold to the open / close determination unit 424.

  Next, the eye opening / closing detection process executed by the eye opening / closing detection system 401 will be described with reference to the flowchart of FIG. This process is started, for example, when the engine of the host vehicle is started, and is ended when the engine is stopped.

  The processing in steps S151 to S153 is the same as the processing in steps S1 to S3 in FIG. 3, and the description thereof will be omitted because it will be repeated. By these processes, the degree of opening and closing of the driver's eyes and the line-of-sight direction are detected based on the face image of the driver's image.

  In step S154, the pupil detection unit 431 detects the pupil. That is, the pupil detection unit 431 detects the driver's pupil in the face image using a predetermined method. The pupil detection unit 431 supplies information indicating the face image and the detection result to the pupil reliability calculation unit 432.

  Note that the method by which the pupil detection unit 431 detects the driver's pupil is not limited to a specific method, and it is desirable to apply a method that can be detected more easily, quickly, and accurately. For example, “Kawaguchi et al., Detection of both eyes from human face by Hough transform and separability filter, IEICE Transactions, Japan, IEICE, 2001, Vol. J84-D-II, No. It is possible to apply the method disclosed in “10, pages 2190-2200”.

  In step S155, the pupil reliability calculation unit 432 calculates the reliability of the pupil detection result. That is, the pupil reliability calculation unit 432 calculates the reliability of the detection result of the driver's pupil by the pupil detection unit 431 using a predetermined method. The pupil reliability calculation unit 432 supplies information indicating the calculated reliability to the threshold setting unit 423.

  Note that the method of calculating the reliability of the pupil detection result by the pupil reliability calculation unit 432 is not limited to a specific method, and it is desirable to apply a method that can be detected more easily, quickly, and accurately. For example, the pupil reliability calculation unit 432 performs template matching on the face image area including the pupil detected by the pupil detection unit 431, calculates the similarity, and uses the calculated similarity as the reliability. Further, for example, the pupil reliability calculation unit 432 calculates the circularity of the pupil region of the face image detected by the pupil detection unit 431 by the following equation (2), and the calculated circularity is used as the reliability. .

Circularity = 4π × S / L ² (2)
Here, S is the area of the pupil region, and L is the perimeter of the region.

  In step S156, the threshold setting unit 423 determines whether or not the reliability of the pupil detection result exceeds a predetermined threshold. If it is determined that the reliability of the pupil detection result exceeds the predetermined threshold, the process proceeds to step S157.

  In step S157, the threshold setting unit 423 sets a determination threshold based on the driver's line-of-sight direction, similar to the processing of the threshold setting unit 123 in step S4 of FIG. Note that the process of step S157 is performed by determining that the reliability of the detection result of the driver's eye direction is high when the reliability of the detection result of the driver's pupil exceeds a predetermined threshold. is there. Thereafter, the process proceeds to step S159.

  On the other hand, if it is determined in step S156 that the reliability of the pupil detection result is equal to or less than the predetermined threshold, the process proceeds to step S159.

  In step S159, as in the process of step S107 in FIG. 7, the determination threshold is set to a value when the driver's line of sight is facing the front. Note that the process in step S159 is performed when the reliability of the detection result of the driver's pupil is equal to or less than a predetermined threshold value, and the reliability of the detection result of the driver's gaze direction is determined to be low. . Thereafter, the process proceeds to step S159.

  The processing in steps S159 to S162 is the same as the processing in steps S5 to S8 in FIG. 3, and the description thereof will be omitted because it will be repeated. Note that the opening / closing of the eyes of the driver is determined by these processes. Thereafter, the process returns to step S151, and the processes after step S151 are executed.

  In this way, when the reliability of the detection result of the driver's line-of-sight direction is low, erroneous detection of the opening / closing of the driver's eyes is suppressed, and the detection accuracy is improved.

  In the above description, in the eye opening / closing detection system 301 in FIG. 6 and the eye opening / closing detection system 401 in FIG. 8, an example in which the determination threshold is set by the same method as the eye opening / closing detection system 101 in FIG. 2 is shown. However, the determination threshold may be set by a method similar to that of the eye opening / closing detection system 201 of FIG. That is, for example, when it is determined that the reliability of the detection result of the driver's gaze direction is low based on the change amount of the driver's gaze direction or the reliability of the detection result of the driver's pupil, the driver The determination threshold may be set using the correction amount when the line of sight of the vehicle faces the front direction of the host vehicle.

  In addition, when it is determined that the reliability of the detection result of the driver's line of sight is low, a determination threshold is set to a value when the driver's line of sight is facing a predetermined direction other than the front as necessary. You may make it do.

  Note that the present invention can also be applied to, for example, detecting the opening and closing of the eyes of a person other than the driver.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 10 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are connected to each other by a bus 504.

  An input / output interface 505 is further connected to the bus 504. The input / output interface 505 includes an input unit 506 made up of a keyboard, mouse, microphone, etc., an output unit 507 made up of a display, a speaker, etc., a storage unit 508 made up of a hard disk or nonvolatile memory, and a communication unit 509 made up of a network interface. A drive 510 for driving a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 501 loads the program stored in the storage unit 508 to the RAM 503 via the input / output interface 505 and the bus 504 and executes the program, for example. Is performed.

  The program executed by the computer (CPU 501) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 511 that is a package medium including a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 508 via the input / output interface 505 by attaching the removable medium 511 to the drive 510. Further, the program can be received by the communication unit 509 via a wired or wireless transmission medium and installed in the storage unit 508. In addition, the program can be installed in the ROM 502 or the storage unit 508 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in the present specification, the term “system” means an overall apparatus composed of a plurality of apparatuses and means.

  Furthermore, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

It is a figure for demonstrating the change of the eye open / closed degree by a gaze direction. It is a block diagram which shows 1st Embodiment of an eye opening / closing detection system. It is a flowchart for demonstrating 1st Embodiment of an eye opening / closing detection process. It is a block diagram which shows 2nd Embodiment of an eye opening / closing detection system. It is a flowchart for demonstrating 2nd Embodiment of an eye opening / closing detection process. It is a block diagram which shows 3rd Embodiment of an eye opening / closing detection system. It is a flowchart for demonstrating 3rd Embodiment of an eye opening / closing detection process. It is a block diagram which shows 4th Embodiment of an eye opening / closing detection system. It is a flowchart for demonstrating 4th Embodiment of an eye opening / closing detection process. And FIG. 11 is a block diagram illustrating an example of a configuration of a computer.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Eye opening / closing detection system 111 Imaging device 112 Eye opening / closing detection device 121 Opening / closing degree detection part 122 Gaze direction detection part 123 Threshold setting part 124 Opening / closing determination part 201 Eye opening / closing detection system 211 Imaging device 212 Eye opening / closing detection apparatus 221 Opening / closing degree detection part 222 Gaze direction detection unit 224 Open / close determination unit 231 Initial threshold setting unit 232 Threshold correction unit 301 Eye open / close detection system 311 Imaging device 312 Eye open / close detection device 321 Open / closed degree detection unit 322 Gaze direction detection unit 323 Threshold setting unit 324 Open / close determination unit 331 Line of sight Direction holding unit 401 Eye open / close detection system 411 Imaging device 412 Eye open / close detection device 421 Open / closed degree detection unit 422 Gaze direction detection unit 423 Threshold setting unit 424 Open / close determination unit 431 Pupil detection unit 432 Pupil reliability calculation unit

Claims (7)

In a detection device that detects opening and closing of a human eye,
An open / closed degree detecting means for detecting an open / closed degree indicating the degree of opening of the person's eyes based on an image of a person's face;
Gaze direction detection means for detecting the gaze direction of the person based on the image;
Threshold setting means for setting a determination threshold used for determination of opening and closing of eyes based on the detected direction of the line of sight of the person;
An opening / closing determination means for determining opening / closing of the human eye by comparing the degree of opening / closing with the determination threshold. The threshold setting means obtains the correction amount for the detected direction of the person's line of sight based on predetermined data indicating a relationship between the direction of the line of sight and a correction amount for correcting the determination threshold, and the correction The detection device according to claim 1, wherein the determination threshold is set by correcting a preset value using an amount. The detection device according to claim 1, wherein the threshold value setting unit sets the determination threshold value for the detected direction of the person's line of sight based on predetermined data indicating a relationship between the direction of the line of sight and the determination threshold value. The threshold setting means obtains a change amount of the person's line-of-sight direction based on a detection result by the line-of-sight direction detection means, and when the change amount is within a predetermined range, The determination threshold is set based on a direction, and when the amount of change exceeds the range, the determination threshold is set to a value when the person's line of sight faces a predetermined direction. Detection device. Pupil detection means for detecting the human pupil in the image;
Pupil reliability calculation means for calculating the reliability of the detection result of the human pupil, and
The threshold setting means, when the reliability exceeds a predetermined threshold, sets the determination threshold based on the detected direction of the line of sight of the person, and when the reliability is equal to or lower than a predetermined threshold, The detection device according to claim 1, wherein the determination threshold value is set to a value when the person's line of sight faces a predetermined direction. A detection device that detects the opening and closing of the human eye,
Based on an image obtained by imaging the person's face, the degree of opening / closing indicating the degree of opening of the person's eyes is detected,
Detecting the direction of the person's line of sight based on the image;
Based on the detected direction of the line of sight of the person, a determination threshold value used for determination of eye opening and closing is set,
A detection method including a step of determining opening / closing of the human eye by comparing the degree of opening / closing with the determination threshold. On the computer,
Based on an image obtained by imaging the person's face, the degree of opening / closing indicating the degree of opening of the person's eyes is detected,
Detecting the direction of the person's line of sight based on the image;
Based on the detected direction of the line of sight of the person, a determination threshold value used for determination of eye opening and closing is set,
A program for executing processing including a step of determining opening / closing of the human eye by comparing the degree of opening / closing with the determination threshold.

Images