JP2021160439A - Head-up display device - Google Patents

Abstract

To provide a vehicular display device capable of stably detecting a position of a pupil of a user even when external light is made incident.SOLUTION: A head-up display device 1 projects display light L1 onto a front windshield 2 of a vehicle, generates and displays a virtual image V by the display light L1 reflected by the front windshield 2, projects infrared light L2 onto a driver D of the vehicle, takes an image of the driver D, and detects a position of a pupil of the driver D on the basis of the taken image. When it is difficult to detect the position of the pupil on one eye E of the right and left eyes of the driver D, a control part 10 estimates the position of the pupil on the one eye E on the basis of the information on positional relationships between the pupil of the driver D and a feature point. When it is determined that the position of the pupil has moved in a predetermined range while the position of the feature point on the other eye E of the right and left eyes of the driver D is maintained, the control unit estimates the position of the pupil on the one eye E on the basis of the information on the positional relationships and the information on the movement.SELECTED DRAWING: Figure 1

Description

The present invention relates to a head-up display device capable of displaying a virtual image on a front windshield, a combiner, or the like of a vehicle and detecting the position of a user's pupil.

A head-up display device that creates and displays a virtual image by the display light reflected by the reflected light-transmitting member by superimposing it on the actual scene (scenery in front of the vehicle) that passes through the reflected light-transmitting member such as the front windshield or combiner of the vehicle. Contributes to safe and comfortable vehicle operation by providing information desired by the user as a virtual image while suppressing the movement of the line of sight of the user such as the driver of the vehicle as much as possible.

In addition, the head-up display device irradiates the user with infrared light to image the user, and detects the position of the eyes such as the user's pupil based on the captured image to look aside or take a nap. There is something to help grasp the state of the user.

For example, the head-up display device described in Patent Document 1 is formed by reflecting visible light emitted from a display means toward a user by a combiner member to form a display image, and infrared rays are directed toward the user. The infrared irradiation means for irradiating the light, the mirror member that reflects the visible light emitted from the display means toward the combiner member and transmits the infrared rays reflected by the user and the combiner member, and the infrared rays transmitted through the mirror member are sensed. It is provided with a plurality of imaging means for imaging the user from different directions and an image processing means for calculating the position of the user's eyes based on the image captured by the imaging means, and the position of the user's eyes can be determined. It is possible to calculate with high accuracy.

Japanese Unexamined Patent Publication No. 2008-126984

By the way, when strong external light such as sunlight or street light is incident on the head-up display device, the noise of the external light is added to the captured image, and there is a problem that the detection of the position of the user's eyes fails. In the head-up display device described in Patent Document 1, when an infrared component of external light passes through a mirror member and is incident on an imaging means, an overexposed noise region due to saturation or the like is formed in the captured image, and this noise region is formed. Even if the captured image does not necessarily overlap with the position of the user's eyes (the position of the image of the eyes), the detection of the position of the eyes may be hindered.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a vehicle display device capable of stably detecting the position of a user's pupil even when external light is incident.

In order to solve the above problems, the present invention projects display light onto a reflective transmissive member provided in a vehicle, superimposes the display light on the actual scene transmitted through the reflective transmissive member, and reflects the display light on the reflective transmissive member. A virtual image is generated and displayed by light, infrared light is projected onto the user of the vehicle to image the user, and the positions of the pupils of the left and right eyes of the user are detected based on the captured image. A possible head-up display device, a pupil detecting means for detecting the position of the pupil in the captured image and a feature point detecting means for detecting the position of the feature point on the contour of the user's eye in the captured image. When it is difficult for the pupil detecting means to detect the position of the pupil for one of the left and right eyes of the user, the one eye is based on the information regarding the positional relationship between the pupil and the feature point of the user. It is provided with a pupil estimation means for estimating the position of the pupil, and the position of the pupil is within a predetermined range while the position of the feature point is maintained for the other left and right eyes of the user by the pupil detecting means and the feature point detecting means. When it is determined that the pupil has moved within, the pupil estimating means estimates the position of the pupil with respect to the one eye based on the information regarding the positional relationship and the information regarding the movement.

Here, "when it is difficult for the pupil detecting means to detect the position of the pupil for one of the left and right eyes of the user" does not mean only when the pupil detecting means fails to detect the position of the pupil. This also includes the case where a predetermined condition is satisfied, which makes it difficult to detect the position of the pupil due to the incident of external light or the like.

Further, the predetermined range may be a range in which the virtual image enters a field of view including at least a discriminative field of view of the other eye, and the positional relationship is the range of the other eye detected by the pupil detecting means. It may be the positional relationship between the position of the pupil and the position of the feature point for the other eye detected by the feature point detecting means, and the feature point may be the outer corner or the inner corner of the eye of the user. ..

Further, the head-up display device according to the present invention includes face orientation detecting means for detecting the orientation of the user's face, and the pupil estimation means is for the user's face detected by the face orientation detecting means. It is determined from the orientation that the user is not facing the virtual image, and the position of the pupil with respect to the other eye detected by the pupil detecting means or the position of the user detected by the face orientation detecting means. When it is determined from the orientation of the face that the user is not visually recognizing the virtual image, the position of the pupil is estimated for the left and right eyes of the user when the user is visually recognizing the virtual image in the orientation of the face. You may.

According to the head-up display device according to the present invention, the position of the user's pupil can be stably detected even when external light is incident.

It is explanatory drawing which shows the head-up display device which concerns on embodiment for carrying out an invention. It is explanatory drawing which shows the image taken by the driver by the head-up display device of FIG. 1 together with a feature point. It is explanatory drawing which shows the mode that the outside light invades into the head-up display device of FIG. It is explanatory drawing which shows the captured image which included the noise region by an external light. It is explanatory drawing which shows the captured image which the position of the pupil of a driver has moved. It is a flow chart which shows the estimation process of the pupil position by the control part of the head-up display device of FIG. It is explanatory drawing which shows the state which a driver does not face a virtual image and does not visually recognize a virtual image in a plan view.

A mode for carrying out the present invention will be described with reference to the drawings.

As shown in FIG. 1, the head-up display device (HUD) 1 according to the present embodiment is provided below the front windshield 2 of the vehicle, and a part of the front windshield 2 is a visible light L. Project ₁ The display light L ₁ is reflected by the front windshield 2 to generate a virtual image V, and causes the driver D of the vehicle to visually recognize the virtual image V by superimposing it on the actual scene transmitted through the front windshield 2.

Further, the HUD 1 has a DMS (driver monitoring system) function of monitoring the state of the driver D, _{projects infrared light L 2} onto the driver D to image the driver D, and operates based on the captured image. The position of the pupil C (see FIG. 2) of the eye E of the person D can be detected.

Specifically, the HUD 1 is covered with a housing 3 formed of a black ABS resin or the like to prevent the intrusion of external light and is partitioned from the outside, and the housing 3 is a transparent resin such as polycarbonate (not shown). A translucent portion 4 covered with is formed. A display unit 5, a folding mirror 6, a concave mirror 7, an infrared light irradiation unit 8, a camera 9, and a control unit 10 are held and housed inside the housing 3.

Here, the display unit 5 is provided with a light source and a liquid crystal panel made of a chip-type light emitting diode, and the liquid crystal panel two-dimensionally modulates the emitted light of the light source to obtain visible light (display light L). _{Although 1} ) is projected and displayed, a reflective device such as a DMD (Digital Micromirror Display) may be used for the display unit 5. The folding mirror 6 is formed by depositing a metal such as aluminum on a resin such as polycarbonate molded so as to have a flat portion, and simply reflects light. The concave mirror 7 is formed by depositing a metal such as aluminum on a resin such as polycarbonate molded so as to have a concave portion, and has a property of magnifying and reflecting visible light and transmitting infrared light.

The infrared light irradiation unit 8 is provided on the back side of the concave mirror 7 (opposite the light transmitting portion 4 and the folding mirror 6 with respect to the concave mirror 7), and emits infrared light (near infrared rays) emitted by a light source composed of a light emitting diode. Irradiate toward 7. The camera 9 includes an image pickup element having sensitivity to infrared light in a wavelength band emitted from the infrared irradiation unit 8 and a lens capable of transmitting infrared light and forming an image on the image pickup device, and provides a near-infrared image. To shoot.

The control unit 10 is composed of a microprocessor, a memory, and various electronic components for operating them, a board, and a case, and is a display unit so as to appropriately display the image of the HUD1 based on the vehicle information and the input information of the driver D. 5 is controlled.

Further, as shown in FIG. 2, the control unit 10 detects the positions of the pupils C of the left and right eyes E of the driver D (for example, the center position of the iris I) based on the contrast of the image P captured by the camera 9. and detects the position of the midpoint C ₁ of the pupil C of the left and right as the viewpoint position. The detected viewpoint position can be used for detecting the state of the driver D (looking aside, taking a nap, etc.) and controlling the display of the HUD1.

_{Further, the control unit 10 has the contour F 1} (including the tip F ₂ of the jaw) of the face F of the driver D, the left and right ears Y, and the left and right ears Y based on the contrast of the image P captured by the camera 9. Center Y ₁ , left and right eyebrows B, left and right eyebrows B center B ₁ , between eyebrows Q, left and right eye E contours E ₁ (including outer corners E ₂ and inner _{corners E 3} ), nose N contour N ₁ (left and right) of including edge _{N 2} nose.), right and left nose edge _{N 2} of the center _{N 3,} holes _{N 4} of the right and left nose, the center _{N 5} holes _{N 4} of the right and left nose, nose _{N 6,} nose tip _{N 7} , human in R, contour M ₁ of the mouth M _(including left and right corners of the mouth M _2.), and detects the position of the feature point T such centers M ₃ of the right and left corners of the mouth M _2, the position of the detected pupil C The positional relationship with the position of the detected feature point T is stored in the internal memory (hereinafter, the positional relationship stored in the control unit 10 is also referred to as a "reference positional relationship").

_{In the HUD 1} , the display light L 1 from the display unit 5 is reflected by the folding mirror 6, then enlarged and reflected by the concave mirror 7, passes through the light transmitting portion 4, and is projected onto the front windshield 2. _{The display light L 1} projected on the front windshield 2 is enlarged and reflected toward the driver D, generates a virtual image V, and is displayed on the driver D overlaid on the actual scene transmitted through the front windshield 2.

On the other hand, the infrared light L ₂ from the infrared light irradiation unit 8 passes through the concave mirror 7, passes through the translucent portion 4, and is projected onto the front windshield 2, and is projected onto the front windshield 2 by the front windshield 2 toward the driver D. It is reflected and irradiates the driver D. Then, when reflected by the driver D, _{a part of the infrared light L 2} follows the opposite path, and the driver D is imaged by _{the infrared light L 2} transmitted through the concave mirror 7 and incident on the camera 9. The captured image P is input to the control unit 10. This imaging is performed periodically or irregularly during the display of the virtual image V, and here, a moving image at a regular frame rate is performed while the virtual image V is displayed.

Control unit 10, the captured image P of the driver D is input, detects the position of the pupil C of the left and right eyes E of the driver D, the position of the midpoint C ₁ of the left and right pupils C viewpoint position Detect as. However, originally, although it should captured image P of the driver D as shown in FIG. 2 is obtained, as shown in FIG. 3, the infrared component windshield of intense external light L ₃ of sunlight 2. When the light-transmitting portion 4 and the concave mirror 7 are transmitted and incident on the camera 9, an overexposed noise region S as shown in FIG. 4 is generated in the captured image P, and the pupil C of the eye E of the driver D (FIG. In 4, it may interfere with the detection of the position of the pupil C) of the right eye E.

Therefore, when it is difficult for the control unit 10 to detect the position of the pupil C for one of the left and right eyes E of the driver D due to the presence of the noise region S in the captured image P, the control unit 10 has the pupil C and the eyes of the driver D. Based on the information regarding the positional relationship of the feature points T on the contour E ₁ of E, the position of the pupil C is estimated for one eye E.

Then, with respect to the other eye E on the left and right of the driver D, the position of the feature point T is maintained, and the position of the pupil C is a virtual image V (hereinafter, also referred to as “HUD image”), which is at least a discriminative visual field of the other eye E. When it is determined that the eye has moved within the range including the visual field (here, the effective visual field), the control unit 10 has the pupil C for one eye E based on the above information on the positional relationship and the information on the movement. Estimate the position of.

For example, the control unit 10 may refer to the captured image P (hereinafter, also referred to as “first captured image P”) at a certain point in time when the driver D is imaged from the front (when the ignition of the vehicle is turned on or when the display of the HUD image is started). ), The relationship between the positions of the pupil C, the outer corner of the eye E ₂ and the inner _{corner of the eye E 3} is stored as a reference positional relationship, and then the captured image P (hereinafter, also referred to as “second captured image P”) is captured. eye E of the driver D is detectable position of the pupil C in, the right eye E is the position of the pupil C by noise area S by the external light L ₃ becomes undetectable, the reference positional relationship, as well, the right eye E _{Based on the position of the outer corner E 2 and} the position of the inner corner E ₃ detected in the above, the position of the pupil C is estimated for the right eye E.

However, when the driver D of those that contain the HUD image effective visual field of the driver D is not looking at the front, as shown in FIG. 5, the outer corner of the eye E ₂ and inner corner of the left eye E in the second captured image P each position of E ₃ while being maintained substantially the same as the position of the outer corner of the eye E ₂ and inner corner of the eye E ₃ of the left eye E in the first captured image P, the position of the pupil C of the left eye E in the second captured image P is , The HUD image is moving from the first captured image P within the range within the effective field of view of the left eye E, and is offset from the position of the pupil C of the left eye E in the first captured image P.

In such a case, the reference positional relationship (relationship between the positions of the pupil C, the outer corner E ₂ and the inner corner E ₃ of the left eye E in the first captured image P) stored by the control unit 10 is applied to the right eye E. alone will, because originally would estimate the pupil C of the right eye E that should be displaced by an amount similar to the same direction and the left eye E (distance), as shown with the pupil C ₀ in FIG. 5, control The unit 10 estimates the position of the pupil C with respect to the right eye E based on not only the information regarding the reference positional relationship but also the information regarding the movement of the position of the pupil C of the left eye E. This estimation corrects the reference positional relationship based on the displacement of the position of the pupil C of the left eye E (for example, the displacement direction and the amount of displacement of the pupil C from the first captured image P to the second captured image P). The corrected reference positional relationship may be applied to the right eye E, or the reference positional relationship may be applied to the right eye E as it is, and then the application result is corrected according to the displacement of the position of the pupil C of the left eye E. You may.

Further, the control unit 10 detects the direction of the face F of the driver D, determines from the detected face direction of the driver D that the driver D does not face the HUD image, and detects the other side. If it is determined from the position of the pupil C with respect to the eye E or the detected face orientation of the driver D that the driver D is not viewing the HUD image, the driver D visually recognizes the HUD image in that face orientation. Estimate the positions of the left and right pupils C at the time.

When the process of estimating the position of the pupil C by the control unit 10 is shown in FIG. 6, the control unit 10 acquires the position information of the feature point T of the face F of the driver D and the pupil C (step 1 (in FIG. 6). Described as "S.1". The same applies hereinafter.)), And confirm whether or not the positions of the pupils C of both the left and right eyes can be obtained (step 2). If YES in step 2, the control unit 10 ends the process because the position of the pupil C has already been detected and does not need to be estimated.

If NO in step 2, the control unit 10 confirms whether or not the position of the pupil C of either the left or right eye can be acquired (step 3), and if NO in step 3, the control unit 10 is left or right. Since the position of the pupil C cannot be obtained for both eyes and it is difficult to continue the estimation process, the process is terminated.

If YES in step 3, the control unit 10 specifies the orientation of the driver D's face F from the position of the feature point T (step 4: the control unit 10 itself does not specify the orientation of the face F, and the illustration is omitted. A specific result may be received from another ECU), and it is determined whether or not the face F of the driver D faces the HUD image (step 5).

If YES in step 5, it is difficult to detect the position of the pupil C for one of the left and right eyes (the eye in which the pupil C is hidden in the noise region S) E of the driver D in the captured image P, while the driver D Since it is considered that both the left and right eyes E are viewing the HUD image, the control unit 10 has the other left and right eyes of the driver D detected from the captured image P (the pupil C is not hidden in the noise region S). For eye) E, the relationship between the positions of pupil C, outer corner E ₂ and inner corner E ₃ is stored as a reference positional relationship (step 6), and the reference positional relationship and the outer corner E ₂ detected for one eye E are stored. based on the positions of and inner corner of the eye E _3, to estimate the position of the pupil C for one eye E (step 7).

If NO in step 5, the control unit 10 looks at the driver D's line of sight from the position of the pupil C with respect to the other eye E detected in step 1 and / or the direction of the driver D's face F specified in step 4. It is specified (step 8), and it is determined whether or not the driver D is visually recognizing the HUD image (step 9). Specifically, for example, the control unit 10 identifies a virtual straight line extending in the same direction as the face direction from the detected position of the pupil C as the line of sight of the driver D, and the driver D assumed from that line of sight. If the HUD image is included in the discriminant field of view, it is determined that the driver D is visually recognizing the HUD image.

_{If YES in step 9, the driver D refers to the outer corner E 2} and the inner corner E ₃ of the pupil C detected in step 1 because the face F does not face the HUD image but the line of sight is looking at the HUD image. The relative position is determined if the control unit 10 remembers the reference positional relationship when the face F faces the HUD image (for example, while the control unit 10 repeats the estimation process shown in FIG. 6). , If the reference positional relationship is memorized in step 6 of the previous estimation process), the reference positional relationship is deviated. Therefore, the control unit 10 detects the offset of the position of the pupil C with respect to the other eye E of the driver D in which the pupil C has been detected, and obtains not only the information regarding the reference positional relationship but also the information regarding the offset. Based on this, the position of the pupil C is estimated for one eye E of the driver D in which the pupil C has not been detected (step 10).

If NO in step 9, as shown in FIG. 7, the driver D does not face the HUD image and does not visually recognize the HUD image. This is a temporary situation such as when checking the side or the rear when turning left or right of the vehicle, and it is considered that the driver D immediately turns his face F to the front and visually recognizes the HUD image. First, the line of sight starts to move toward the front, and then the action of changing the direction of the face F follows. The control unit 10 estimates in advance the moving position (predicted viewpoint position) of the pupil C for the left and right eyes E in preparation for the next viewing of the HUD image (step 11), and corrects the position of the display of the HUD image (AR superimposition). By executing warping control according to the display position) and the viewpoint position, it is possible to control the display without discomfort when the line of sight returns to the HUD image.

Specifically, when it is determined in step 9 that the driver D is not visually recognizing the HUD image, the control unit 10 assumes that the driver D sees the center of the HUD image (center of the angle of view), and the pupil C For the other eye (left eye in FIG. 7) E of the detected driver D, the pupil position (the position of the pupil _{indicated by the symbol C 0L} in FIG. 7) when looking at the center of the HUD image while keeping the face orientation is estimated. , after calculating the offset between the position of the position and the pupil C _0L of the current pupil C, one of the eyes of the driver D pupil C is not detected by the external light L ₃ of the offset (in Figure 7 right) E Also for one eye E, the pupil position (the position of the pupil indicated by the reference _{numeral C0R in FIG. 7) when looking at the center of the HUD image while keeping the face orientation is estimated.} Such an estimation does not necessarily have to be performed with reference to the center of the HUD image, and when the driver D is facing the left side as shown in FIG. 7, the line of sight of the driver D sees the HUD image from the left end. Therefore, in order to more accurately estimate the viewpoint position at which the HUD image is first visually recognized, the left end of the HUD image may be used as a reference.

In the HUD 1 according to the present embodiment, when it is difficult for the control unit 10 to detect the position of the pupil C with respect to one of the left and right eyes E of the driver D, the positional relationship between the pupil C of the driver D and the feature point T. based on (reference position relationship) information about, so to estimate the position of the pupil C for one eye E, if the detection of one or both of the positions of the left and right eyes E are difficult due to the influence of the external light L ₃ also, allows the estimation of the position of the fit E Rarere obtained position of the characteristic point T by imaging, reduces the probability of missed detection of the position of the eye E, the position of the eye E even when external light L ₃ incident Can be detected stably.

Further, when it is determined that the position of the feature point T is maintained and the position of the pupil C is moved within a predetermined range with respect to the other left and right eyes E of the driver D, the control unit 10 provides information on the reference positional relationship. Since the position of the pupil C is estimated for one eye E based on the information on the movement and the movement thereof, even if the driver D moves the pupil C without moving the face F too much and swings the line of sight from the front to the left, right, up and down. , The position of the pupil C can be accurately estimated for one eye E, and the probability of failure to detect the position of the eye E can be reduced.

Further, the control unit 10 determines that the driver D is facing sideways (horizontally) or vertically (vertically) from the direction of the driver D's face F and is not facing the HUD image. If it is determined from the position of the pupil C with respect to the other eye E of the driver D and / or the direction of the face F of the driver D that the driver D is not visually recognizing the HUD image, the driver D is the face. Since the positions of the left and right pupils C when visually recognizing the HUD image in the orientation are estimated, the driver D can then control the display of the HUD image by using the predicted viewpoint position determined by the estimated position of the pupil C. A smooth display change can be realized when the line of sight is directed toward the HUD image.

Although the embodiments for carrying out the present invention have been illustrated above, the embodiments of the present invention are not limited to those described above, and may be appropriately modified without departing from the spirit of the invention.

For example, in the above embodiment, an example in which the outer corner E ₂ and the inner corner E ₃ of the eye are used as the feature points T for estimating the position of the pupil C has been described, but other desirable points _{E 1} on the contour E 1 of the eye E have been described. The position of the pupil C may be estimated from a plurality of feature points T.

1 Head-up display device 2 Front windshield (reflection and translucent member)
9 Camera 10 Control unit (pupil detection means, feature point detection means, pupil estimation means, face orientation detection means)
C Pupil D Driver (user)
E Eye E ₁ Contour E ₂ Outer corner E ₃ Inner eye F Face L ₁ Display light L ₂ Infrared light P Captured image T Feature point V Virtual image

Claims (5)

The display light is projected onto the reflected light-transmitting member provided on the vehicle, and a virtual image is generated and displayed by the display light reflected by the reflected light-transmitting member by superimposing the display light on the actual scene transmitted through the reflected light-transmitting member. A head-up display device capable of projecting infrared light onto a vehicle user to image the user and detecting the positions of the pupils of the left and right eyes of the user based on the captured image.
A pupil detecting means for detecting the position of the pupil in the captured image, and
A feature point detecting means for detecting the position of a feature point on the contour of the user's eye in the captured image, and
When it is difficult for the pupil detecting means to detect the position of the pupil for one of the left and right eyes of the user, the pupil of the one eye is based on the information regarding the positional relationship between the pupil and the feature point of the user. Equipped with a pupil estimation means for estimating the position,
When it is determined by the pupil detecting means and the feature point detecting means that the position of the feature point is maintained for the other left and right eyes of the user and the position of the pupil is moved within a predetermined range, the pupil estimation is performed. The means is a head-up display device that estimates the position of the pupil with respect to the one eye based on the information regarding the positional relationship and the information regarding the movement thereof. The head-up display device according to claim 1, wherein the predetermined range is a range in which the virtual image enters a field of view including at least a discriminative field of view of the other eye. The positional relationship is the positional relationship between the position of the pupil with respect to the other eye detected by the pupil detecting means and the position of the feature point with respect to the other eye detected by the feature point detecting means. The head-up display device according to claim 1 or 2. The head-up display device according to any one of claims 1 to 3, wherein the feature point is the outer or inner corner of the eyes of the user. A face orientation detecting means for detecting the orientation of the user's face is provided.
The pupil estimating means determines from the orientation of the user's face detected by the face orientation detecting means that the user does not face the virtual image, and the other one detected by the pupil detecting means. When it is determined from the position of the pupil with respect to the eyes or the orientation of the user's face detected by the face orientation detecting means that the user is not visually recognizing the virtual image, the user is the orientation of the face. The head-up display device according to any one of claims 1 to 4, wherein the position of the pupil is estimated for the left and right eyes of the user when the virtual image is visually recognized.

Images