JP7019394B2 - Visual target detection device, visual target detection method, and program - Google Patents

Description

The present invention relates to a technique for detecting an object visually recognized by a driver.

For example, there are known techniques for determining whether a driver has confirmed safety when changing lanes while driving a vehicle, and techniques for detecting inattentive driving and issuing an alarm. Patent Document 1 discloses an inattentive alarm device that gives an appropriate alarm according to the details of the inattentive state of the driver. This inattentive alarm device determines whether the driver's gaze or inattentiveness depends only on the line-of-sight direction or the face-facing direction. Set longer than when relying solely on.

Japanese Unexamined Patent Publication No. 2011-164712

As seen in Patent Document 1 described above, it has been known to use the driver's face orientation and line of sight detected from an image taken by a camera as a method of detecting a visual state by the driver. The method of detecting the visual recognition state using the line of sight has high accuracy in detecting the direction in which the driver is looking, but the range in which the line of sight can be detected by one camera is narrow. On the contrary, the method of detecting the visual recognition state using the face orientation has a wide detection range, but the accuracy of detecting the visual recognition state is low. For example, research by the inventors so far has shown that when a driver looks at a speedometer or rear-view mirror, he hardly moves his face, only his eyes. Also, when looking at the same object, how much the face orientation changes depends on the individual or the situation.

In view of the above background, an object of the present invention is to provide a visual object detection device capable of appropriately detecting an object visually recognized by a driver.

The visual object detection device of the present invention includes a camera that captures the driver, a face orientation detection unit that detects the driver's face orientation from an image captured by the camera, and a driver's face orientation detection device from an image captured by the camera. An output that outputs information indicating a visual target estimated by the visual target estimation unit, a visual target estimation unit that estimates the visual target that the driver is visually recognizing from the line of sight or the face direction, and a line-of-sight detection unit that detects the line of sight. When the detection accuracy of the line of sight detected by the line-of-sight detection unit is equal to or higher than a predetermined threshold value, the visual-view target estimation unit estimates the visual target based on the line of sight detected by the line-of-sight detection unit. If the detection accuracy of the line of sight is smaller than a predetermined threshold value, the visual target is estimated based on the face orientation detected by the face orientation detecting unit.

With this configuration, when the detection accuracy of the driver's line of sight is high, the visual target is estimated by the line of sight, and when the detection accuracy of the line of sight is low, the visual target is estimated by the face of the driver. It is possible to achieve both height and a wide detectable range.

In the visual object detection device of the present invention, the visual object estimation unit is a network model in which the face direction, the line of sight, and the visual object are nodes, and the relationship between the face direction and the visual object, and the line of sight and the visual object is expressed by probabilistic propagation. When estimating the visual target from the face orientation by reading from the storage unit, the evidence is not input to the line-of-sight node, but the detected facial orientation data is input to the face-facing node, and the visual target is displayed from the line-of-sight. In the case of estimation, the gaze target may be estimated by inputting the detected line-of-sight data into the line-of-sight node without inputting evidence into the face-facing node. With this configuration, the visual target can be estimated appropriately.

In the visual object detection device of the present invention, the visual object estimation unit may estimate the visual object based on the face orientation and the line of sight when the detection accuracy of the line of sight is smaller than a predetermined threshold value. With this configuration, it is possible to accurately estimate the visual target by using both the line-of-sight and face orientation data.

The visual object detection device of another aspect of the present invention includes a camera that captures the driver, a face orientation detecting unit that detects the face orientation of the driver from the image captured by the camera, and an image captured by the camera. A line-of-sight detection unit that detects the driver's line of sight from the camera, a line-of-sight estimation unit that estimates the driver's line of sight from the face direction, a view target estimation unit that estimates the visual target that the driver is viewing from the line of sight, and the above-mentioned visual recognition. The visual target estimation unit includes an output unit that outputs information indicating a visual target estimated by the target estimation unit, and the visual target estimation unit is provided with a line-of-sight detection accuracy detected by the line-of-sight detection unit when the detection accuracy is equal to or higher than a predetermined threshold value. The visual target estimation unit estimates the visual target based on the line of sight detected by the line-of-sight detection unit, and if the line-of-sight detection accuracy is smaller than a predetermined threshold, the face detected by the face orientation detection unit. The line of sight is estimated from the direction, and the visual object is estimated based on the line of sight estimated by the line-of-sight estimation unit.

With this configuration, when the detection accuracy of the driver's line of sight is high, the visual target is estimated by the line of sight, and when the detection accuracy of the line of sight is low, the visual target is estimated using the line of sight estimated from the driver's face direction. Therefore, it is possible to achieve both high estimation accuracy and wide detectable range.

In the visual object detection device of the present invention, the visual object estimation unit uses a face direction, a line of sight, and a visual object as nodes, and represents a network model in which the relationship between the face direction and the line of sight and the line of sight and the visual object is expressed by probabilistic propagation. When reading from the storage unit and estimating the visual target from the face orientation, the detected face orientation data is input to the face orientation node, and when estimating the visibility target from the line of sight, the face orientation node is used. The gaze target may be estimated by inputting the detected line-of-sight data to the line-of-sight node without inputting the evidence. With this configuration, the visual target can be estimated appropriately.

In the visual object detection device of the present invention, the visual object estimation unit has a line of sight estimated from the face orientation detected by the face orientation detection unit and the line of sight detection when the detection accuracy of the line of sight is smaller than a predetermined threshold value. The line of sight detected by the unit may be weighted based on the detection accuracy of the line of sight by the line-of-sight detection unit to estimate the line of sight of the driver, and the line of sight may be estimated using the estimated line of sight. With this configuration, the line of sight can be appropriately estimated based on the detection result of the face orientation and the detection result of the line of sight, and by extension, the visual target can be estimated accurately.

In the visual object detection device of the present invention, the camera may be a single camera that captures the driver from the front. Even in the case of a single camera that can accurately detect the line of sight only within a predetermined range, the configuration of the present invention can appropriately estimate the visual target.

The visual target detection method of the present invention is a method of detecting a visual target visually recognized by the driver by the visual target detection device, the step of the visual target detection device taking a picture of the driver with a camera, and the above-mentioned. A step in which the visual object detection device detects the driver's face orientation from an image taken by the camera, and a step in which the visual object detection device detects the driver's line of sight from an image taken by the camera. Information indicating a visual target estimated by the visual target detection device in a step of estimating a visual target visually recognized by the driver from the line of sight or the face direction, and a step of estimating the visual target by the visual target detection device. The step of estimating the visual object is detected by the step of detecting the line of sight when the detection accuracy of the line of sight detected in the step of detecting the line of sight is equal to or higher than a predetermined threshold value. The visual target is estimated based on the line of sight, and when the detection accuracy of the line of sight is smaller than a predetermined threshold value, the visual target is estimated based on the face orientation detected in the step of detecting the face orientation.

Another aspect of the present invention is a method of detecting a visual object visually recognized by a driver by a visual object detecting device, wherein the visual object detecting device photographs the driver with a camera. A step, a step in which the visual object detection device detects the driver's face orientation from an image taken by the camera, and a step in which the visual object detection device detects the driver's line of sight from an image taken by the camera. Steps, a step in which the visual object detection device estimates the driver's line of sight from the face direction, a step in which the visual object detection device estimates the visual object visually recognized by the driver from the line of sight, and the visual recognition. The step of estimating the visual target includes a step of outputting information indicating the visual target estimated in the step of estimating the target, and in the step of estimating the visual target, the detection accuracy of the line of sight detected in the step of detecting the line of sight is equal to or higher than a predetermined threshold value. In the case of, the visual target is estimated based on the line of sight detected in the step of detecting the line of sight, and when the detection accuracy of the line of sight is smaller than a predetermined threshold value, in the step of detecting the face orientation. The line of sight is estimated based on the detected face orientation, and the visual target is estimated based on the estimated line of sight.

The program of the present invention is a program for detecting a visual object visually recognized by the driver, based on a step of inputting image data of the driver taken by a camera into a computer and the image data. A step of detecting the driver's face orientation, a step of detecting the driver's line of sight based on the image data, a step of estimating a visual object visually recognized by the driver from the line of sight or the face direction, and the visual object. In the step of estimating the visual target, the step of outputting the information indicating the estimated visual target is executed in the step of estimating the visual target, and the detection accuracy of the line of sight detected in the step of detecting the line of sight is equal to or higher than a predetermined threshold. In the case of, the visual object is estimated based on the line of sight detected in the step of detecting the line of sight, and when the detection accuracy of the line of sight is smaller than a predetermined threshold value, in the step of detecting the face orientation. The visual target is estimated based on the detected face orientation.

The program of the present invention is a program for detecting a visual object visually recognized by the driver, and is based on a step of inputting image data of the driver taken by a camera into a computer and the image data. The step of detecting the driver's face orientation, the step of detecting the driver's line of sight based on the image data, the step of estimating the driver's line of sight from the face direction, and the step of visually recognizing the driver from the line of sight. It includes a step of estimating a visual target and a step of outputting information indicating the visual target estimated in the step of estimating the visual target, and the step of estimating the visual target is detected by the step of detecting the line of sight. When the detection accuracy of the line of sight is equal to or higher than a predetermined threshold, the visual target is estimated based on the line of sight detected in the step of detecting the line of sight, and when the detection accuracy of the line of sight is smaller than the predetermined threshold. , The line of sight is estimated based on the face direction detected in the step of detecting the face direction, and the visual object is estimated based on the estimated line of sight.

According to the present invention, it is possible to achieve both high estimation accuracy for estimating a visual object visually recognized by the driver and a wide range in which a visible object can be detected.

It is a figure which shows the structure of the visual object detection device of 1st Embodiment. It is a figure which shows the example of the object which is visually recognized in a car. It is a figure which shows the structure of the visual inspection target estimation part. It is a figure which shows the operation of the visual object detection device of 1st Embodiment. It is a figure which shows the structure of the visual object detection device of 2nd Embodiment. It is a figure which shows the structure of the visual inspection target estimation part. It is a figure which shows the operation of the visual object detection device of 2nd Embodiment.

Hereinafter, the visual object detection device according to the embodiment of the present invention will be described with reference to the drawings.
(First Embodiment)
FIG. 1 is a diagram showing a configuration of a visual object detection device 1 according to the first embodiment. The visual object detection device 1 is a device that detects an object (visual object) visually recognized by the driver of the automobile. By detecting the visual object with the visual object detection device 1, it is determined whether the driver is looking aside, for example, whether the driver has taken appropriate safety actions when changing lanes or turning left or right. You can judge.

The visual object detection device 1 of the first embodiment includes a camera 10, a face orientation detection unit 11, a line-of-sight detection unit 12, a visual target estimation unit 13, and an output unit 14.

The camera 10 is a near-infrared camera, and can stably photograph the driver even when the vehicle interior is dark. The camera 10 is mounted at a position where the driver is photographed from the front, for example, under the meter visor. The camera 10 of the "driver status monitor" developed by Denso Corporation may be used.

The face orientation detection unit 11 has a function of detecting the driver's face orientation based on the driver's face image taken by the camera 10. The line-of-sight detection unit 12 has a function of detecting the driver's line of sight from the image of the driver's eyes taken by the camera 10. The line-of-sight detection unit 12 detects the direction in which the driver's black eyes are facing, and adds the direction of the driver's face to the direction of the line of sight with respect to the driver's face. Detect if it is. When the line-of-sight detection unit 12 detects the line of sight of the driver, the line-of-sight detection unit 12 measures the detection accuracy and obtains a value (quality value) indicating the degree of certainty. Here, the detection accuracy is, for example, a value that indicates the correctness of the detected direction of the line of sight with a probability. It is the accuracy to judge the accuracy of the detection of the line-of-sight direction from the appearance of the white and black parts of the eyeball as a result of the driver's own blinking, eye closure, and the positional relationship between the camera and the eyes.

The visual object estimation unit 13 has a function of estimating an object visually recognized by the driver based on the driver's face orientation and line of sight. The visual target estimation unit 13 reads out the SVM model for estimating the visual target stored in the SVM model storage unit 15, and applies the face orientation data and the line-of-sight data to the SVM model to estimate the visual target. ..

FIG. 2 is a diagram showing an example of an object visually recognized in the vehicle. As an example of the visual object in the vehicle, FIG. 2 shows a window shield display, a speedometer, a car navigation system, a rear-view mirror, a right mirror, a left mirror, a right side, and a left side. FIG. 2 shows an example, and the visual object that can be visually recognized by the visual object detection device 1 is not limited to these.

In FIG. 2, the window shield display, the speedometer, the car navigation system, and the right mirror, which are in the front direction, are objects within a range that can be detected by the line of sight. Then, the entire visual object including the detection range by the line of sight is the detection range by the face orientation. That is, the detection range using the face orientation is wider than the detection range using the line of sight.

FIG. 3 is a diagram showing the configuration of the visual inspection target estimation unit 13. The visual recognition target estimation unit 13 has a noise filter 20, an evaluation unit 21, and a determination unit 22. The noise filter 20 has a function of removing noise included in the face orientation data input from the face orientation detection unit 11 or the line-of-sight data input from the line-of-sight detection unit 12. The evaluation unit 21 evaluates a score indicating the possibility of seeing each visual object from the input face orientation data or line-of-sight data by processing using a multi-class SVM model. As the SVM model, the SVM model read from the SVM model storage unit 15 is used. When it is desired to estimate the visual object from the line of sight, the SVM model learned using the line of sight is read out, and when it is desired to estimate the visual object from the face direction, the SVM model learned using the face direction is used. When it is desired to estimate the visual object based on both the line of sight and the face direction, an SVM model learned using both the line of sight and the face direction is used. The determination unit 22 determines the visual target having the maximum score based on the score of each visual target obtained by the evaluation unit 21, and outputs the visual target data.

The output unit 14 outputs the visibility target data estimated by the visibility target estimation unit 13 to a driving support device such as an inattentive determination device.

Although the configuration of the visual target detection device 1 of the present embodiment has been described above, examples of the hardware of the visual target detection device 1 described above include a CPU, RAM, ROM, hard disk, display, keyboard, mouse, communication interface, and the like. It is a computer equipped with. The above-mentioned program is stored in a RAM or ROM having a module that realizes each function of the face-facing detection unit 11, the line-of-sight detection unit 12, and the visual-view target estimation unit 13, and the program is executed by the CPU. The function of the visual object detection device 1 is realized. Such programs are also included in the scope of the present invention. An example of the output unit 14 is a communication interface.

FIG. 4 is a diagram showing the operation of the visual object detection device 1 according to the first embodiment. The visual object detection device 1 of the first embodiment takes a picture of the driver by the camera 10 (S10). Subsequently, the visual object detection device 1 detects the driver's face orientation by the face orientation detection unit 11 based on the image obtained by shooting (S11), and also detects the driver's line of sight by the line-of-sight detection unit 12. (S12). When the line-of-sight detection unit 12 detects the line of sight of the driver, the line-of-sight detection unit 12 obtains a quality value of the detection accuracy.

The visual recognition target detection device 1 determines whether or not the line-of-sight detection accuracy is equal to or higher than the threshold value based on the quality value of the line-of-sight detection in the line-of-sight detection unit 12 (S13). When the line-of-sight detection accuracy is equal to or higher than the threshold value (YES in S13), the visual target is estimated from the line-of-sight detected by the line-of-sight detection unit 12 (S15). The visual target estimation unit 13 reads the SVM model learned by using the line of sight from the SVM model storage unit 15, and inputs the detected line-of-sight data into the SVM model to estimate the visual target.

When the line-of-sight detection accuracy is not equal to or higher than the threshold value (NO in S13), the visual object is estimated from the face orientation detected by the face orientation detecting unit 11 (S14). The visual target estimation unit 13 estimates the visual target by reading the model learned using the face orientation from the SVM model storage unit 15 and inputting the detected face orientation data into the model. The visual object detection device 1 outputs the data of the visual object estimated by the visual object estimation unit 13 to another device (S16).

The configuration and operation of the visual object detection device 1 according to the first embodiment have been described above. Since the visual target detection device 1 of the first embodiment selects whether to estimate the visual target by the line of sight based on the detection accuracy of the line of sight by the line-of-sight detection unit 12, or to estimate the visual target by the face orientation, the detection accuracy is high. Both height and wide detection range can be achieved.

The visual target detection device 1 of the above-described embodiment has described a configuration in which the visual target is estimated using the detected face orientation data when the line-of-sight detection accuracy is not equal to or higher than the threshold value. If is not equal to or greater than the threshold value, a configuration may be adopted in which the line-of-sight data is applied to the SVM model together with the detected face orientation data to estimate the visual target.

(Second embodiment)
FIG. 5 is a diagram showing the configuration of the visual object detection device 2 according to the second embodiment. The basic configuration of the visual object detection device 2 of the second embodiment is the same as that of the visual object detection device 1 of the first embodiment, but the processing of the visual object by the visual object estimation unit 13 is different. The visual recognition target estimation unit 13 reads out the Bayesian network model stored in the Bayesian model storage unit 16 and estimates the visual recognition target by applying the face orientation data and the line-of-sight data to the read-out model.

FIG. 6 is a diagram showing the configuration of the visual inspection target estimation unit 13. The visual object estimation unit 13 has a probability distribution unit 30 and a Bayesian net estimation unit 31. The probability distribution unit 30 has a function of probabilistically distributing the face orientation data input from the face orientation detection unit 11 or the line-of-sight data input from the line-of-sight detection unit 12. The Bayesian net estimation unit 31 applies probability-distributed face-facing data and line-of-sight data to the Bayesian network model read from the Bayesian model storage unit 16 to estimate the probability of a visual target. The Bayesian network model shown in FIG. 6 is a model in which a face-facing node, a line-of-sight node, and a node to be visually recognized are connected in order.

The visual recognition target estimation unit 13 changes the data applied to the Bayesian network estimation unit 31 depending on the accuracy of the line-of-sight detection by the line-of-sight detection unit 12. Specifically, when the line-of-sight detection accuracy is equal to or higher than a predetermined threshold value, no evidence is input to the face-facing node, and the probability distribution of the detected line-of-sight data is input to the line-of-sight node. As a result, the visual target estimation unit 13 can estimate the visual target based on the line-of-sight data.

If the line-of-sight detection accuracy is not greater than or equal to a predetermined threshold, the probability distribution of the detected face-facing data is input to the face-facing node, and no evidence is input to the line-of-sight data node. The probability distribution of the line-of-sight node is calculated from the probability distribution of the face-facing node, and the visual target can be estimated based on the probability distribution of the line-of-sight node. As a result, one Bayesian network model can estimate the visual target in both the case where the line-of-sight detection accuracy is high and the case where the line-of-sight detection accuracy is low.

FIG. 7 is a diagram showing the operation of the visual object detection device 2 according to the second embodiment. The visual object detection device 2 of the second embodiment takes a picture of the driver by the camera 10 (S20). Subsequently, the visual object detection device 2 detects the driver's face orientation by the face orientation detection unit 11 based on the image obtained by shooting (S21), and also detects the driver's line of sight by the line-of-sight detection unit 12. (S22). When the line-of-sight detection unit 12 detects the line of sight of the driver, the line-of-sight detection unit 12 obtains a quality value of the detection accuracy.

The visual recognition target detection device 2 determines whether or not the line-of-sight detection accuracy is equal to or higher than the threshold value based on the quality value of the line-of-sight detection in the line-of-sight detection unit 12 (S23). When the line-of-sight detection accuracy is equal to or higher than the threshold value (YES in S23), the visual target estimation unit 13 inputs the probability distribution of the detected line-of-sight data to the Bayesian network estimation unit 31's line-of-sight data node, and the visual target is to be visually recognized. Is estimated (S25).

When the line-of-sight detection accuracy is not equal to or higher than the threshold value (NO in S23), the probability distribution of the face orientation data detected by the face orientation detection unit 11 is input to the face orientation data node of the Bayesian net estimation unit 31. The probability distribution of the line-of-sight data is obtained from the probability distribution of the face-facing data (S24), and the visual object is estimated from the probability distribution of the line-of-sight data (S25). The visual object detection device 2 outputs the data of the visual object estimated by the visual object estimation unit 13 to another device (S26).

The configuration and operation of the visual object detection device 2 according to the second embodiment have been described above. The visual target detection device 2 of the second embodiment either estimates the visual target by the line of sight based on the detection accuracy of the line of sight by the line-of-sight detection unit 12, or estimates the visual target based on the line of sight estimated from the face direction. It is possible to achieve both high accuracy and a wide detection range.

In the second embodiment, when the line-of-sight detection accuracy is not equal to or higher than the threshold value (NO in S23), the visual target is estimated based on the line-of-sight estimated from the face orientation data (S24 to S25), but the face orientation. In addition to the line of sight estimated from the data, the line of sight data obtained by the line of sight detection unit 12 may be used to estimate the line of sight of the driver, and the visual target may be obtained based on the estimated line of sight. In this case, as a method of combining the line of sight estimated from the face orientation and the line of sight detected by the line of sight detection unit 12, a configuration in which weighting is performed using the detection accuracy of the line of sight by the line of sight detection unit 12 can be considered. For example, it is assumed that the line-of-sight detection result by the line-of-sight detection unit 12 is 30 degrees to the right, the detection accuracy is 10%, and the line-of-sight estimated from the face orientation data is 40 degrees to the right. In this case, the line-of-sight direction may be estimated as 30 × 0.1 + 40 × (1-0.1) = 3 + 36 = 39 degrees.
With this configuration, the line of sight can be appropriately estimated based on the detection result of the face orientation and the detection result of the line of sight, and by extension, the visual target can be estimated accurately.

Although the visual object detection device of the present invention has been described in detail by taking an embodiment as an example, the visual object detection device of the present invention is not limited to the above-described embodiment. In the above description, as the visual target estimation unit 13, an example in which the SVM model is used in the first embodiment and the Bayesian network model is used in the second embodiment is given, but the visual target in the first embodiment is used. The Bayesian network model may be used in the detection device 1, or the SVM model may be used in the visual target detection device 2 of the second embodiment.

Further, instead of the Bayesian network model used in the second embodiment, a network model in which the relationship between nodes such as a Gaussian network, a Markov chain, and a hidden Markov model is expressed by belief propagation may be used.

1, 2 Visual target detection device 10 Camera 11 Face orientation detection unit 12 Line-of-sight detection unit 13 Visual target estimation unit 14 Output unit 15 SVM model storage unit 16 Bayesian model storage unit 20 Noise filter 21 Evaluation unit 22 Determining unit 30 Probability distribution unit 31 Bayesian Net Estimator

Claims (4)

A camera that shoots the driver,
A face orientation detection unit that detects the driver's face orientation from the image taken by the camera, and
A line-of-sight detection unit that detects the driver's line of sight from the image taken by the camera, and
A line-of-sight estimation unit that estimates the driver's line of sight from the face orientation,
A visual object estimation unit that estimates the visual object that the driver is visually recognizing from the line of sight,
An output unit that outputs information indicating the visual target estimated by the visual target estimation unit, and an output unit that outputs information indicating the visual target.
Equipped with
When the detection accuracy of the line of sight detected by the line-of-sight detection unit is equal to or higher than a predetermined threshold value, the visual-view target estimation unit is a visual target based on the line of sight detected by the line-of-sight detection unit. When the estimation is performed and the detection accuracy of the line of sight is smaller than a predetermined threshold value, the line of sight is estimated from the face direction detected by the face orientation detection unit , and the estimated line of sight and the line of sight detected by the line of sight detection unit are used. In addition, a visual object detection device that estimates the driver's line of sight by weighting based on the detection accuracy of the line of sight, and estimates the visual target using the estimated line of sight . The visual object detection device according to claim 1 , wherein the camera is a single camera that photographs the driver from the front. It is a method of detecting a visual object that the driver is visually recognizing by a visual object detection device.
The step in which the visual object detection device captures the driver with a camera,
A step in which the visual object detection device detects the driver's face orientation from an image taken by the camera, and
A step in which the visual object detection device detects the driver's line of sight from an image taken by the camera.
The step in which the visual object detection device estimates the driver's line of sight from the face direction,
The step in which the visual object detection device estimates the visual object visually recognized by the driver from the line of sight, and
A step of outputting information indicating the estimated visual object in the step of estimating the visual object, and a step of outputting the information indicating the visual object.
Equipped with
In the step of estimating the visual target, when the detection accuracy of the line of sight detected in the step of detecting the line of sight is equal to or higher than a predetermined threshold value, the visual target is based on the line of sight detected in the step of detecting the line of sight. When the estimation is performed and the detection accuracy of the line of sight is smaller than a predetermined threshold value, the line of sight is estimated based on the face direction detected in the step of detecting the face direction, and the estimated line of sight and the line of sight are detected. The line of sight detected in the step is weighted based on the detection accuracy of the line of sight to estimate the line of sight of the driver, and the line of sight is estimated using the estimated line of sight.
Visual target detection method. It is a program for detecting the visual object that the driver is visually recognizing, and it is a program for the computer.
Steps to input the driver's image data taken by the camera,
A step of detecting the driver's face orientation based on the image data,
A step of detecting the driver's line of sight based on the image data,
The step of estimating the driver's line of sight from the face orientation,
The step of estimating the visual object that the driver is visually recognizing from the line of sight,
A step of outputting information indicating the estimated visual object in the step of estimating the visual object, and a step of outputting the information indicating the visual object.
To execute,
In the step of estimating the visual target, when the detection accuracy of the line of sight detected in the step of detecting the line of sight is equal to or higher than a predetermined threshold value, the visual target is based on the line of sight detected in the step of detecting the line of sight. When the estimation is performed and the detection accuracy of the line of sight is smaller than a predetermined threshold value, the line of sight is estimated based on the face direction detected in the step of detecting the face direction, and the estimated line of sight and the line of sight are detected. The line of sight detected in the step is weighted based on the detection accuracy of the line of sight to estimate the line of sight of the driver, and the line of sight is estimated using the estimated line of sight.
program.

Images