JP2022128697A - Sight line detection apparatus - Google Patents

Abstract

To provide a sight line detection apparatus that can provide accurate sight line data of a driver in an on-vehicle environment.SOLUTION: A sight line detection apparatus 1 includes a camera 12 that photographs the face of a driver 11 to acquire facial image, and a processing unit 13 that calculates sight line data of the driver 11 from the facial image. The processing unit 13 includes a sight line data collection mode for storing the sight line data in a storage unit, and a sight line data calibration mode for calibrating the sight line data. The sight line data calibration mode calculates a plurality of high frequency gaze scopes which are frequently gazed among a plurality of gaze scopes set based on in-vehicle information, calculates a sight line distribution based on the plurality of sight line data stored in the storage unit, and specifies an actual sight line position scope corresponding to each of the high frequency gaze scopes from the sight line distribution. The sight line data calibration mode calculates a correction value of the sight ling data based on a relative position of the plurality of high frequency gaze scopes and a relative position of the plurality of actual sight line position scope, and corrects and generates the sight line data based on the calculated correction value.SELECTED DRAWING: Figure 6

Description

The present invention relates to a line-of-sight detection device.

2. Description of the Related Art Conventionally, a line-of-sight estimation device and a line-of-sight detection device for estimating a driver's line of sight by image processing from a moving image of a driver's face have been known (see Patent Documents 1 to 3, for example).

Japanese Patent Application Laid-Open No. 2010-23670 JP-A-2005-261728 Japanese Unexamined Patent Application Publication No. 2008-210239

In order to detect the line of sight of the driver, the conventional line-of-sight detection device needs to accurately detect the center position of the iris and the center point of the eyeball from the face image of the driver. For example, since the center point of the eyeball does not appear visually, it is detected from the visible part of the eyeball or estimated from the center of the outer corner-inner corner of the eye. Therefore, in an in-vehicle environment, it is desired to provide line-of-sight data through accurate line-of-sight detection.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a line-of-sight detection device capable of providing accurate line-of-sight data of a driver in an in-vehicle environment.

In order to achieve the above object, the line-of-sight detection device according to the present invention includes an image acquisition unit that acquires a face image by photographing the face of a driver of a vehicle; and a processing unit that calculates line-of-sight data including the line-of-sight position of the driver, wherein the processing unit has a line-of-sight data collection mode in which the line-of-sight data is stored in a storage unit in association with time series, and the line-of-sight data is corrected. and a line-of-sight data calibration mode, wherein the line-of-sight data calibration mode selects a plurality of high-frequency gaze ranges from among a plurality of gaze ranges during driving set based on the vehicle interior information. calculating a line-of-sight distribution based on a plurality of the line-of-sight data stored in the storage unit; identifying actual line-of-sight position ranges corresponding to each of the high-frequency gaze ranges from the line-of-sight distribution; calculating a correction value for the line-of-sight data based on the relative position of the gaze range and the relative positions of the plurality of the actual line-of-sight position ranges, and correcting and generating the line-of-sight data based on the calculated correction value; It is characterized by

INDUSTRIAL APPLICABILITY The line-of-sight detection device according to the present invention has the effect of being able to provide accurate line-of-sight data of a driver in an in-vehicle environment.

FIG. 1 is a schematic diagram showing an application example of a line-of-sight detection device according to an embodiment. FIG. 2 is a flowchart showing an overview of a line-of-sight detection algorithm executed by the line-of-sight detection device according to the embodiment. FIG. 3A is a schematic diagram showing the relationship between a three-dimensional (3D) model of an eyeball and a two-dimensional (2D) image, and FIG. It is a schematic diagram. FIG. 4 is a flowchart showing an outline of a mode switching algorithm executed by the line-of-sight detection device according to the embodiment. FIG. 5 is a schematic diagram showing a state of the vehicle when mode switching performed by the line-of-sight detection device according to the embodiment is applied. FIG. 6 is a flow chart diagram showing an example of a gaze data calibration mode algorithm executed by the gaze detection device according to the embodiment. FIG. 7A is a schematic diagram showing an overview of multiple high-frequency gaze ranges in the line-of-sight detection device according to the embodiment, and FIG. It is a schematic diagram which shows the outline|summary of a center position. FIG. 8 is a schematic diagram showing an example of correspondence between line-of-sight positions and line-of-sight distributions in the line-of-sight detection device according to the embodiment. FIG. 9 is a graph diagram showing an example of a line-of-sight detection result by the line-of-sight detection device according to the embodiment. FIG. 10A is a graph diagram showing an outline of one polygon used when calculating a correction value in the line-of-sight detection device according to the embodiment, and FIG. FIG. 10 is a graph diagram showing an overview of the other polygon used when calculating the correction value;

A line-of-sight detection device according to an embodiment of the present invention will be described in detail below with reference to the drawings. In addition, this invention is not limited by embodiment shown below. Components in the following embodiments include components that can be easily assumed by those skilled in the art, or components that are substantially the same. Also, the constituent elements in the following embodiments can be omitted, replaced, and changed in various ways without departing from the scope of the invention.

[Embodiment]
A line-of-sight detection device 1 of this embodiment shown in FIG. The line-of-sight detection device 1 includes a camera 12 and a processing section 13 .

The camera 12 is an example of an image acquisition unit or an image acquisition device, and captures the face of the driver 11 of the vehicle 2 to acquire face images including the driver's face in time series. The camera 12 is, for example, a digital camera having a CCD (Charge-Coupled Device) image sensor, a CMOS (complementary metal oxide semiconductor) image sensor, or the like. The camera 12 is installed, for example, in the meter unit in front of the driver's seat or above the column cover in order to photograph the face of the driver 11 sitting in the driver's seat in the passenger compartment 10 of the vehicle 2 . The camera 12 acquires a facial image including the eye area 11a of the driver 11 . The camera 12 is electrically connected to the processing unit 13 and outputs to the processing unit 13 various kinds of information such as moving image data (or still image data) corresponding to the obtained face image and the shooting time of the face image.

The processing unit 13 is an example of a processing device, and performs image processing on the face image acquired by the camera 12 to calculate and output line-of-sight data. The sight line data is information indicating the direction of the sight line 11b from the eye area 11a in the face image. Specifically, the line-of-sight data includes the line-of-sight angle of the driver 11 obtained by image processing the face image, the gaze position, and the photographing time of the face image. The line-of-sight data is used by an in-vehicle device (for example, a device that recognizes whether the driver 11 is dozing off and issues a warning, etc.) to recognize whether or not the driver 11 is moving the line of sight for safety confirmation. can be done.

The processing unit 13 has, for example, a processing circuit (not shown) that implements various processing functions in the line-of-sight detection device 1 . A processing circuit is implemented by, for example, a processor. A processor means a circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array). The processing unit 13 implements each processing function by, for example, executing a program read from a storage circuit (storage unit) (not shown). Further, the processing unit 13 can store a plurality of pieces of line-of-sight data in association with the photographing time of the face image in the storage unit, and can read the line-of-sight data stored in the storage unit and perform image processing.

The processing unit 13 has a line-of-sight data collection mode and a line-of-sight data calibration mode. The line-of-sight data collection mode is a mode in which line-of-sight data is stored in a storage unit in association with time series. The line-of-sight data calibration mode is a mode for correcting the line-of-sight data using the line-of-sight data stored in the storage unit in the line-of-sight data collection mode. The processing unit 13 switches between a line-of-sight data collection mode and a line-of-sight data calibration mode. For example, the processing unit 13 switches to the line-of-sight data collection mode when the vehicle 2 is running, and switches to the line-of-sight data calibration mode when the vehicle 2 is stopped. The processing unit 13 determines whether the vehicle 2 is in a running state or in a stopped state, for example, based on vehicle state information acquired from an ECU (Electronic Control Unit) of the vehicle 2 side. The vehicle state information indicates, for example, whether or not an accelerator pedal or brake pedal (not shown) of the vehicle 2 has been operated. It should be noted that the determination as to whether the vehicle 2 is in the running state may be made using a running sensor or the like, in addition to whether or not the accelerator pedal or the brake pedal of the vehicle 2 is being operated.

Next, a line-of-sight detection algorithm executed by the line-of-sight detection device 1 will be described with reference to FIG. In the processing shown in FIG. 2, each step is sequentially performed by the processing unit 13 executing a program read from the storage unit.

The camera 12 constantly captures an image of an area including the face of the driver 11 at regular intervals and outputs an image signal. In step S<b>12 , the processing unit 13 acquires one frame of video signals from the camera 12 .

In step S13, the processing unit 13 performs conversion (preprocessing) of the data format of the image, including grayscaling. For example, for each pixel position within a frame, 8-bit data representing the brightness in a gradation range of "0 to 255" is arranged vertically and horizontally according to the scanning direction within the frame at the time of shooting. (2D) Generate array image data.

In step S14, the processing unit 13 performs face detection using, for example, the "Viola-Jones method", and extracts a rectangular area including a face from the two-dimensional image data of one frame. That is, the face region is extracted using a detector that is characterized by the shadow difference of the face and created by learning using “Boosting”. The technique of the "Viola-Jones method" is disclosed, for example, in the following documents. "P. viola and M.J. Jones, Rapid Object Detection Using a Boosted Cascade of Simple Features, IEEE CVPR (2001)."

In step S15, the processing unit 13 detects the eye region from the data of the rectangular region of the face extracted in step S14, for example, using the Viola-Jones method described above.

In step S<b>16 , the processing unit 13 estimates the eyeball center when the line of sight is detected by a model-based method using a 3D eyeball model, which will be described later. Here, it is assumed that the center coordinates of the rectangular region of the eye detected in step S15 are the center of the eyeball. For example, a method of determining the center of the eyeball based on the positions of the outer corner and the inner corner of the eye, or a method of estimating the skeleton from the feature points of the face and simultaneously calculating the position of the center of the eyeball may be used.

In step S17, the processing unit 13 applies a template matching method to the data of the rectangular region of the eye detected in step S15, and performs rough search for the iris (pupil or iris). Specifically, the coordinates of the image center (the center of the black circle) of the black circle image with the highest likelihood are matched to the image obtained by binarizing the eye image cut out around the eye. The center position of the iris in the image is set, and the radius of the black circle image with the highest likelihood is set as the iris radius in the eye image. Note that the process of step S17 is performed to roughly estimate the center position and radius of the iris in the eye image.

In step S18, the processing unit 13 uses the center position and radius of the iris searched in step S17 and applies a particle filter technique to detect the center position and radius of the iris with higher accuracy.

In step S19, the processing unit 13 obtains numerical data of eyeball center coordinates and iris center coordinates for one frame image by the processing of steps S13 to S18 described above, and outputs the numerical data. The direction of the line of sight can be specified by the coordinates of the center of the eyeball and the coordinates of the center position of the iris. Real-time line-of-sight detection is realized by repeating the loop-like processing of steps S11 to S20 until the image output by the camera 12 is interrupted. When calculating the eyeball rotation angle from the coordinates on the image plane using these data, conversion using the eyeball model shown in FIG. 3A is performed.

In the eyeball 3D model 30 shown in FIG. 3A, the eyeball is a sphere represented by an eyeball center 31 and an eyeball radius R. Also, on the surface of this eyeball, there is a circular iris 32 that constitutes the iris, and a circular pupil 33 is located in the center of the iris 32 . The direction of the line of sight can be specified as the direction from the eyeball center 31 to the center of the iris 32 or the pupil 33, and is determined by the rotation angle yaw with respect to the reference direction in the horizontal plane and the rotation angle pitch (pitch) with respect to the reference direction in the vertical direction. can be represented. Also, the center coordinates of the iris 32 or the pupil 33 can be represented by the eyeball radius R, yaw, and pitch when the eyeball center 31 is used as a reference. On the other hand, since the image captured by the camera 12 represents a two-dimensional plane, when applying the two-dimensional image 34 captured by the camera 12 to the eyeball 3D model 30, two-dimensional/three-dimensional Mutual conversion is required. For example, the conversion is performed using the following formula.

X=−R×cos(pitch)×sin(yaw) (1)
Y=R×sin(pitch) (2)
X: distance in the x direction from the eyeball center 31 on the two-dimensional image plane Y: distance in the y direction from the eyeball center 31 on the two-dimensional image plane

Next, the process of searching for black eyes from a face image will be described with reference to FIG. 3B. In step S41, the processing unit 13 uses the result of step S15 in FIG. 2 to extract the eyes and their surrounding rectangular regions from the entire two-dimensional image data of one frame to acquire data D41.

In step S42, the processing unit 13 generates data D42 by binarizing the data D41 acquired in step S41 so that the gradation for each pixel is only binary black/white. In contrast, template matching in step S17 in FIG. 2 is performed. That is, an image of a black circle similar to the shape of the iris is used as a template, and while this template is scanned on the image of the data D42, the position of the iris having the most similar characteristics is searched for, and the position and the iris are searched for. Specify the radius or diameter of

In step S43, the processing unit 13 performs Sobel filter processing on the eye data D41 acquired in step S41. Specifically, the eye data D41 is sequentially scanned in the horizontal direction from left to right, and black (gradation value: 0) is output in portions where there is no luminance change, and white is output as the gradient of luminance change increases. (Gradation value: 255) to detect edges. As a result, edge-extracted eye image data D43 is obtained.

In step S44, the processing unit 13 performs particle filter processing on the eye image data D43 obtained in step S43, starting from the approximate iris position obtained in step S42. The black eye detection by the particle filter is performed in the order of [1] to [4] below.

[1] The processing unit 13 searches for a rough iris position by template matching. [2] The processing unit 13 calculates an eyeball rotation angle from the iris position in [1] based on the eyeball 3D model 30 .
[3] The processing unit 13 scatters particles for iris ellipse candidate sampling in the vicinity of the approximate eyeball rotation angle obtained in [2] above.
[4] The processing unit 13 calculates the likelihood at each scattering position from the edge image of the Sobel filter, and estimates the state.

In the vicinity of the eyeball rotation angle of [3] above, multiple patterns are created by adding random numbers to the yaw and pitch values of the eyeball 3D model shown in FIG. 3A. At this time, the projected iris ellipse candidate on the image from each rotation angle can be confirmed by the overlap of the circle 44a of D44. The likelihood calculation of [4] is performed using the values obtained by sampling and adding the gradation values from the circumference of each candidate ellipse.

Next, a mode switching algorithm executed by the line-of-sight detection device 1 will be described with reference to FIG. In the process shown in FIG. 4, each step is sequentially performed by the processing unit 13 executing a program read from the storage unit in response to power-on (for example, ignition switch (IGN) ON).

In step S51, the processing unit 13 performs initial setting processing. In the initial setting process, reading of information of the camera 12 necessary for line-of-sight detection and confirmation of normal operation of the line-of-sight detection device 1 are performed.

At step S52, the processing unit 13 determines whether the vehicle 2 is in a running state. When it is determined that the vehicle 2 is in the running state, the process proceeds to step S53 and switches to the line-of-sight data collection mode. On the other hand, when it is determined that the vehicle 2 is stopped, the mode is switched to the line-of-sight data calibration mode in step S55. That is, the processing unit 13 switches between the line-of-sight data collection mode and the line-of-sight data calibration mode according to the running state of the vehicle 2 .

At step S54, the processing unit 13 determines whether or not the IGN is turned off. The processing unit 13 determines whether the IGN is in the ON state or the OFF state, for example, based on the IGN state information acquired from the ECU on the vehicle 2 side. If IGN is ON in step S54, the process returns to step S52. On the other hand, if the IGN is turned off, this processing ends.

In this embodiment, as shown in FIG. 5, when the vehicle 2 is traveling in a certain section, it shifts to the line-of-sight data collection mode. do. For example, when the vehicle 2 is traveling toward an intersection, the line-of-sight data collection mode is entered. On the other hand, when the vehicle 2 stops in response to the red light at the intersection and is in a stopped state, the line-of-sight data collection mode is temporarily stopped and the line-of-sight data calibration mode is entered. When the vehicle 2 becomes running again in response to the green light at the intersection, the line-of-sight data calibration mode is temporarily stopped and the line-of-sight data collection mode is entered.

Next, the algorithm of the line-of-sight data calibration mode executed by the line-of-sight detection device 1 will be described with reference to FIG. In the processing shown in FIG. 6, each step is sequentially performed by the processing unit 13 executing a program read from the storage unit in response to power-on (for example, IGN ON).

In step S61, the processing unit 13 reads the vehicle interior information, and calculates a plurality of high frequency gaze ranges based on the vehicle interior information. The vehicle interior information indicates, for example, the driver's 11 eyepoint and the design values of the vehicle 2 . The eyepoint may be unique to the driver 11 or may be a reference eyepoint that is a vehicle design standard. The eye point is presumed to be located within the so-called eye range of the vehicle 2 . The eye range corresponds to an area where the viewpoint of the driver 11 is located, which is predetermined according to the vehicle. The eye range statistically represents the distribution of eye positions of the driver 11 in the vehicle 2. For example, when the driver 11 is seated in the driver's seat, a predetermined percentage (eg, 95%) of the drivers corresponds to the region containing the positions of the eyes of The design values of the vehicle 2 are, for example, the design values of the inside of the passenger compartment 10 . Since the vehicle interior information is determined by the vehicle type of the vehicle 2, it is registered in advance in the storage unit within the processing unit 13. FIG. Note that the vehicle interior information may be registered in advance in an ECU or the like on the vehicle 2 side, and the processing unit 13 may acquire the information from the ECU as necessary.

The line of sight of the driver 11 while driving is generally focused on objects in front of the vehicle 2, such as lanes, signs, traffic lights, other vehicles, pedestrians, and the like. At this time, the gazing position (gazing direction) of the driver 11 is included in the driving visual field range 21 assumed in front of the vehicle 2, as shown in FIG. 7A, for example. In the present embodiment, the driving visual field range 21 is set as one of the high-frequency gaze ranges in which the gaze positions of the driver 11 are concentrated. The driving visual field range 21 is generally a range that can be seen by the driver 11 while driving without moving his or her eyes. , left 12°, right 11°, up 6°, down 5°.

In addition, the line of sight of the driver 11 during driving is directed to the left and right mirrors of the vehicle 2 and the meters of the vehicle 2 (eg, speedometer, tachometer, navigation, etc.). Although the left and right mirrors and meters are outside the driver's field of vision, the driver 11 needs to view them periodically to obtain information necessary for driving. Therefore, gaze positions other than the driving visual field range 21 include a gaze range 22 corresponding to the left mirror, a gaze range 23 corresponding to the right mirror, and a gaze range 24 corresponding to the meters. In this embodiment, the high frequency gaze range is defined as a gaze range 22 corresponding to the left mirror, a gaze range 23 corresponding to the right mirror, and a gaze range 24 corresponding to the meters. Among the multiple high-frequency gaze ranges, the driver's 11 most gazed range is the driving visual field range 21 . This is followed by a gaze range 23 corresponding to the right mirror, a gaze range 22 corresponding to the left mirror, and a gaze range 24 corresponding to the meters.

In step S61, the processing unit 13 calculates the relative positions of multiple high-frequency gaze ranges. First, the processing unit 13 calculates the central position of each high-frequency fixation range based on the eye point and the design values of the vehicle 2 . The relative position of the driving visual field range 21 in the horizontal direction of the vehicle 2 is, as shown in FIG. is calculated by the angle θd (the included angle) formed by a virtual line 26 passing through the center position 21a of . The relative position of the gaze range 23 corresponding to the right mirror in the horizontal direction of the vehicle 2 is, as shown in FIG. It is calculated by the angle θR (the side of the included angle) formed by The relative position of the gaze range 22 corresponding to the left mirror in the horizontal direction of the vehicle 2 is, as shown in FIG. It is calculated by the angle θL (the side of the included angle) formed by In this way, the processing unit 13 calculates the relative position of each high-frequency gaze range in the horizontal direction of the vehicle 2 based on the calculated angles θd, θR, and θL. The processing unit 13 also calculates the relative position of each high-frequency gaze range in the vertical direction of the vehicle 2 by the same method as described above.

In step S62, the processing unit 13 calculates the distribution of line-of-sight positions (hereinafter also referred to as "line-of-sight distribution") based on the plurality of pieces of line-of-sight data stored in the line-of-sight data collection mode (FIG. 8). The line-of-sight distribution includes line-of-sight position distribution number: dist, distribution width: σ, median: g(yaw, pitch). First, among the line-of-sight distributions obtained from a plurality of line-of-sight data, the range 41 with the largest number of distributions is defined as the first distribution Gd1, the range 42 with the second largest number of distributions is defined as the second distribution Gd2, and the range 43 with the second largest number of distributions is defined as Gd2. Let the third distribution be Gd3, . . . (FIG. 9). Next, the processing unit 13 calculates the median value and the distribution range (standard deviation σ) of each of the first distribution Gd1, the second distribution Gd2, the third distribution Gd3, . . . The second distribution Gd2 and the third distribution Gd3 have distribution numbers in the range of 20% to 5% of the first distribution Gd1, and distributions with a distribution number of 5% or less are discarded.

In step S63, the processing unit 13 obtains the correspondence relationship between the multiple high-frequency gaze ranges calculated in step S61 and the multiple real line-of-sight position ranges calculated in step S62. In this embodiment, the first distribution Gd1 (41), the second distribution Gd2 (42), the third distribution Gd3 (43), . First, the processing unit 13 associates the first distribution Gd1 with the driving visual field range 21 because the range having the largest number of distributions among the visual line distributions is the first distribution Gd1. Next, in order to check the correspondence between the second distribution Gd2, the third distribution Gd3, . . . and the gaze ranges 22, 23, . A distance (Euglid distance) from the median (center position) of the line-of-sight distribution Gdn (n=2, 3, 4, . . . ) is calculated by the following equation (3). As a result, the one with the smallest distance d is matched.

Distance d={(Gdn.yaw−R.yaw) ² +(Gdn.pitch−R.pitch) ² } ^1/2 (3)

The above formula (3) corresponds to the distance d between the line-of-sight distribution Gdn and the gaze range 23 corresponding to the right mirror. The processing unit 13 uses Equation (3) to calculate the distance d between the line-of-sight distribution Gdn and the gaze ranges 22, 23, .

In step S64, the processing unit 13 determines whether correction value calculation is possible. False detection of the line of sight of the driver 11 or unexpected viewing behavior during driving may occur, and the line of sight may be distributed to portions other than the driving visual field, the right mirror, the left mirror, and the meters. In such a case, the accuracy of correcting the line-of-sight data is greatly reduced, so whether or not the line-of-sight distribution is as expected in the driving section is checked before calculating the correction value. The following two items are checked.

Standard deviation value: σ
The processing unit 13 compares the distribution width of the driving visual field range 21 and the distribution width of the first distribution Gd1. If the distribution width differs more than a certain width (threshold value), the processing unit 13 determines that the correction value is not appropriate and that the correction value cannot be calculated. In addition, since the angle ranges corresponding to the gaze range 23 corresponding to the right mirror and the gaze range 22 corresponding to the left mirror are determined from the design values of the vehicle 2 (ranges in which the left and right mirrors can be visually recognized), they are For reference, it is determined whether or not the correction value is appropriate.

Distance: d
The processing unit 13 determines from the distance d between the center positions 21a, 22a, and 23a of the driving visual field range 21 and the gaze ranges 22 and 23, and the corresponding first distribution Gd1, second distribution Gd2, and third distribution Gd3. . If the distance d is too far from the fixed distance (threshold value), the processing unit 13 determines that the correction value is not appropriate and that the correction value cannot be calculated. In this embodiment, as a guideline, it is within 5 degrees. The standard of 5 degrees is the same as the deviation between the visual axis and the optical axis of the human eyeball. In addition, the target of 5 degrees can also be determined with reference to the detection accuracy (measurement accuracy) of a line-of-sight detection device (such as a line-of-sight measurement system) that is actually used. Note that the distance d is obtained by the above formula (3).

If the correction value calculation is not possible as a result of the determination in step S64, this process ends. On the other hand, if correction value calculation is possible, the process proceeds to step S65.

In step S<b>65 , the processing unit 13 calculates the correction value of the line-of-sight data based on the relative positions of the multiple high-frequency gaze ranges and the relative positions of the multiple actual line-of-sight position ranges. First, the processing unit 13 determines the shape of a triangle 51 ((B) in FIG. 10) having the vertices at the center positions 21a, 22a, and 23a of the driving visual field range 21 and the gaze ranges 22 and 23, and the corresponding first Comparison with the shape of a triangle 52 ((A) in FIG. 10) having vertices at the real center positions 41a, 43a, and 42a of the distribution Gd1 (41), the second distribution Gd2 (42), and the third distribution Gd3 (43) do. The shape change of the triangles 51 and 52 can be calculated by the following formula (4). It is assumed that one point D(x, y) of the triangle 51 becomes Gd1(x', y') after the shape change. At this time, the shape change is represented by the following equation (4) using the transformation matrix.

However, the correction matrix A is as follows.

In order to obtain the components of the correction matrix A, three or more corresponding points are required. A triplicate with Gd3 is used.

In step S66, the processing unit 13 stores the calculated correction matrix A in the storage unit as correction values for line-of-sight data.

The processing unit 13 uses the correction matrix A to calculate the line-of-sight angle G(y, p) detected in the line-of-sight data collection mode using the following equation (5). The line-of-sight angle Gc(y', p') can be obtained by Equation (5).

The processing unit 13 corrects the line-of-sight data stored in the storage unit using the correction value of the line-of-sight data, and outputs the corrected line-of-sight data to the in-vehicle device. By using the corrected line-of-sight data, the in-vehicle device can, for example, accurately grasp the visual recognition behavior of the driver 11 .

The line-of-sight detection apparatus 1 described above includes a camera 12 that captures the face of the driver 11 of the vehicle 2 to acquire a face image, and an image processing of the acquired face image to detect a line-of-sight including the line-of-sight position of the driver 11 . and a processing unit 13 for calculating data. The processing unit 13 has a line-of-sight data collection mode in which line-of-sight data is stored in the storage unit in a time-series manner, and a line-of-sight data calibration mode in which the line-of-sight data is corrected. In the line-of-sight data calibration mode, a plurality of high frequency gaze ranges are calculated from among a plurality of gaze ranges during driving set based on the vehicle interior information of the vehicle 2 . In the line-of-sight data calibration mode, the line-of-sight distribution is calculated based on a plurality of pieces of line-of-sight data stored in the storage unit, and from the line-of-sight distribution, the actual line-of-sight position range corresponding to each high-frequency gaze range is specified. In the line-of-sight data calibration mode, line-of-sight data correction values are calculated based on the relative positions of multiple high-frequency gaze ranges and the relative positions of multiple actual line-of-sight position ranges, and the line-of-sight data is corrected based on the calculated correction values. Correct and generate.

With the above configuration, the reference high-frequency gaze range (21, 22, 23) on the side of the vehicle 2 and the actual line-of-sight position range (Gd1, Gd2, Gd3) based on the actual line-of-sight distribution of the driver 11 are compared. It is possible to clarify how much the direction of the line of sight deviates. As a result, since an overall correction value can be derived from a plurality of line-of-sight data, it is possible to provide accurate line-of-sight data of the driver in the in-vehicle environment. Further, the calibration work of the line-of-sight detection device, which has been performed at the start of measurement of line-of-sight data, becomes unnecessary.

Further, the line-of-sight detection device 1 switches to the line-of-sight data collection mode when the vehicle 2 is running, and switches to the line-of-sight data calibration mode when the vehicle 2 is stopped. Although it is desired to use line-of-sight data when the vehicle 2 is in a running state, the performance of the processing unit 13 needs to be improved if the line-of-sight data is to be calibrated while the vehicle is running. In addition, when collecting line-of-sight data when the vehicle 2 is in a stopped state, the accuracy of the line-of-sight data decreases when, for example, the driver 11 looks away. . Therefore, by switching to the line-of-sight data collection mode when the vehicle 2 is in a running state, and by switching to the line-of-sight data calibration mode when the vehicle 2 is in a stopped state, efficient and accurate data can be obtained without increasing the performance of the processing unit 13 . can provide accurate line-of-sight data.

Further, in the line-of-sight detection device 1, the plurality of high-frequency gaze ranges correspond to a driving visual field range corresponding to a visible range in front of the vehicle 2, a visible range corresponding to the left mirror of the vehicle 2, and a right mirror of the vehicle 2. The visible range is included. The processing unit 13 corrects the line-of-sight data by comparing the triangle connecting the center positions of the multiple high-frequency gaze ranges with the triangle connecting the actual center positions calculated based on the line-of-sight data within the plurality of actual line-of-sight position ranges. Calculate the value. As a result, since it is possible to calculate the correction value of the line-of-sight data with a simple process, it is possible to efficiently calculate the correction value of the line-of-sight data even in a short-time stop state while ensuring accuracy.

[Modification]
In the above embodiment, the processing unit 13 calculates the triangle connecting the center positions 21a, 22a, and 23a of the three high-frequency gaze ranges, and the actual center position calculated based on the line-of-sight data within the three actual line-of-sight position ranges. Although the correction value of line-of-sight data is calculated by comparing with the triangle connecting 41a, 42a, and 43a, it is not limited to this. For example, among multiple high-frequency gaze ranges, a polygon connecting the center positions of four or more high-frequency gaze ranges in descending order, A configuration may be employed in which a correction value for line-of-sight data is calculated by comparing with a polygon connecting real center positions 41a, 42a, and 43a based on data. Further, it is preferable that the number of gaze ranges to be compared among a plurality of high-frequency gaze ranges and the number of actual line-of-sight position ranges be the same.

Further, in the above embodiments and modified examples, the processing circuit is described as one in which each processing function is realized by a single processor, but the present invention is not limited to this. The processing circuit may implement each processing function by combining a plurality of independent processors and having each processor execute a program. Moreover, the processing functions of the processing circuit may be appropriately distributed or integrated in a single or a plurality of processing circuits. Further, the processing functions of the processing circuit may be implemented entirely or in part by a program, or may be implemented by hardware such as wired logic.

In addition, in the above embodiments and modifications, the line-of-sight detection device 1 is applied to a vehicle 2 such as an automobile, but it is not limited to this, and may be applied to ships, aircraft, etc. other than the vehicle 2, for example. Moreover, although the line-of-sight detection device 1 is divided into the camera 12 and the processing unit 13, it is not limited to this, and may be configured integrally.

The program executed by the processor described above is pre-installed in a memory circuit or the like and provided. This program may be recorded in a computer-readable storage medium and provided as a file in a format that can be installed in these devices or in a format that can be executed. Also, this program may be provided or distributed by being stored on a computer connected to a network such as the Internet and downloaded via the network.

Reference Signs List 1 line-of-sight detection device 2 vehicle 10 vehicle interior 11 driver 12 camera 13 processing unit 21 driving visual field range 22, 23 gaze range 21a, 22a, 23a center position

Claims (3)

an image acquisition unit that acquires a face image by photographing the face of the driver of the vehicle;
a processing unit that performs image processing on the acquired face image to calculate line-of-sight data including at least the line-of-sight position of the driver;
The processing unit is
a line-of-sight data collection mode in which the line-of-sight data is associated with time series and stored in a storage unit;
a line-of-sight data calibration mode for correcting the line-of-sight data;
The line-of-sight data calibration mode includes:
calculating a plurality of high-frequency gaze ranges with high frequency among a plurality of gaze ranges during driving set based on the vehicle interior information;
calculating a line-of-sight distribution based on the plurality of line-of-sight data stored in the storage unit, and specifying actual line-of-sight position ranges corresponding to each of the high-frequency gaze ranges from the line-of-sight distribution;
calculating a correction value for the line-of-sight data based on the relative positions of the plurality of high-frequency gaze ranges and the relative positions of the plurality of actual line-of-sight position ranges;
correcting and generating the line-of-sight data based on the calculated correction value;
A line-of-sight detection device characterized by: The processing unit is
when the vehicle is in a running state, switching to the line-of-sight data collection mode;
Switching to the line-of-sight data calibration mode when the vehicle is stationary;
The line-of-sight detection device according to claim 1 . The plurality of high-frequency gaze ranges are
A driving visual field range corresponding to a visual range in front of the vehicle, a visual range corresponding to the left mirror of the vehicle, and a visual range corresponding to the right mirror of the vehicle,
The processing unit is
correcting the line-of-sight data by comparing a polygon connecting the center positions of the plurality of high-frequency gaze ranges with a polygon connecting the actual center positions calculated based on the line-of-sight data within the plurality of the actual line-of-sight position ranges; calculate the value,
The line-of-sight detection device according to claim 1 or 2.

Images