JP2022025741A - Viewpoint position detection system - Google Patents

Abstract

To provide a viewpoint detection system with which it is possible to detect the viewpoint position of users even when a grayscale difference in an imaged image is insufficient.SOLUTION: A viewpoint detection system pertaining to the present invention comprises: a camera for imaging a driver D of a vehicle by infrared light projected to the driver D; and a control unit for detecting the positions of feature points of the driver D on the basis of a grayscale difference in the imaged image of the camera, detecting the positions of left and right irises IL, IR of the driver D in the imaged image on the basis of the detected feature points, and detecting the position of viewpoint IC of the driver D on the basis of the detected positions of the irises IL, IR . When it is difficult to detect the position of a feature point such as an inner canthus E1R, an outer canthus E2R or an upper eyelid E3R and difficult to detect the position of one of the irises IL, IR, the control unit generates a gamma corrected image Pγ in which the gamma value of the imaged image is adjusted, and detects the position of the feature point on the basis of a grayscale difference in the gamma corrected image Pγ.SELECTED DRAWING: Figure 5

Description

The present invention relates to a viewpoint position detection system that is used for a front windshield of a vehicle, a head-up display device that displays a virtual image on a combiner, and the like, and detects the position of the viewpoint of a user such as a driver of the vehicle.

A head-up display device that creates and displays a virtual image by the display light reflected by the reflected light-transmitting member, which is superimposed on the actual scene (scenery in front of the vehicle) that passes through the reflected light-transmitting member such as the front windshield and combiner of the vehicle. Contributes to safe and comfortable vehicle operation by providing information desired by the user as a virtual image while suppressing the movement of the line of sight of the user such as the driver of the vehicle as much as possible.

In addition, the head-up display device irradiates the user with infrared light to image the user, and detects the position (viewpoint position) of the user's eyes or the like based on the captured image to look aside. There is something to help grasp the user's condition such as falling asleep.

For example, the head-up display device described in Patent Document 1 is formed by reflecting visible light emitted from a display means toward a user by a combiner member to form a display image, and infrared rays are directed toward the user. The infrared irradiation means that irradiates the infrared rays, the mirror member that reflects the visible light emitted from the display means toward the combiner member and transmits the infrared rays reflected by the user and the combiner member, and the infrared rays transmitted through the mirror member. It is provided with a plurality of imaging means for imaging the user from different directions and an image processing means for calculating the position of the user's eyes based on the image captured by the imaging means, and the position of the user's eyes can be determined. It is possible to calculate with high accuracy.

Japanese Unexamined Patent Publication No. 2008-126984

By the way, in the viewpoint detection system used in such a head-up display device, the user's characteristic points (eyes themselves may be used as characteristic points) based on the gradation difference (contrast) in the captured image (captured image data). By detecting the contour of the face and the position of the ears, nose tip, eyebrows, chin, etc.), the positions of the left and right eyes of the user are detected, and the viewpoint position is detected. When strong external light such as sunlight or street light is incident on the imaging means, or as a result of adjusting the exposure according to the strong external light in the imaging means, a part of the user's face in the captured image becomes bright. It may be too dark (overexposed) or too dark (underexposed). Then, there is a problem that a sufficient gradation difference for detecting the feature point cannot be obtained and the viewpoint cannot be detected.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a viewpoint detection system capable of detecting a user's viewpoint position even when the gradation difference in a captured image is not sufficient.

In order to solve the above problems, the viewpoint detection system according to the present invention has a gradation difference between an image pickup means for capturing an image of the user by infrared light projected on the user of the vehicle and an image captured by the image pickup means. Based on the feature point detecting means for detecting the position of the feature point of the user and the feature point detected by the feature point detecting means, the positions of the left and right eyes of the user in the captured image are detected. A viewpoint position detecting means for detecting the viewpoint position of the user based on the detected positions of the left and right eyes is provided, and the feature point detecting means is difficult to detect the position of the feature point and the viewpoint. When it is difficult for the position detecting means to detect the position of one of the left and right eyes, a gamma-corrected image in which the gamma value of the captured image is adjusted is generated, and the feature is based on the gradation difference in the gamma-corrected image. It is characterized by detecting the position of a point.

When it is difficult for the feature point detecting means to detect the position of the feature point and it is difficult for the viewpoint position detecting means to detect the position of one of the left and right eyes, it is difficult to detect the position of the captured image. The gamma-corrected image may be generated by adjusting the gamma value of the region including the feature points.

For example, when the gamma value of the captured image is α, the feature point detecting means adjusts the gamma value to be larger than α within a range where the input of the captured image is less than a predetermined threshold, and the gamma correction is performed. An image may be generated, and further, the gamma-corrected image may be generated by adjusting the gamma value to be smaller than α in the range where the input of the captured image is equal to or larger than the threshold value.

Further, the feature point detecting means does not generate the gamma-corrected image when it is difficult to detect the position of the feature point and it is difficult for the viewpoint position detecting means to detect the positions of both the left and right eyes. May be good.

According to the viewpoint detection system according to the present invention, the viewpoint position of the user can be detected even when the gradation difference in the captured image is not sufficient.

It is explanatory drawing which shows the head-up display apparatus to which the viewpoint detection system which concerns on embodiment for carrying out an invention is applied. It is explanatory drawing which shows the image taken by the driver by the head-up display apparatus of FIG. 1 together with the feature point. It is explanatory drawing which shows the state which the outside light invades into the head-up display device of FIG. It is explanatory drawing which shows the captured image which it is difficult to detect the feature point by the outside light together with the feature point. It is explanatory drawing which shows the gamma correction image which adjusted the gamma value of the captured image of FIG. 4 together with a feature point. It is explanatory drawing which shows the example of the generation method of the gamma-corrected image. It is explanatory drawing which shows the other example of the generation method of the gamma-corrected image. It is explanatory drawing which shows still another example of the generation method of the gamma-corrected image.

A mode for carrying out the present invention will be described with reference to the drawings.

As shown in FIG. 1, the head-up display device (HUD) 1 to which the viewpoint detection system according to the present embodiment is applied is provided below the front windshield 2 of the vehicle and is a part of the front windshield 2. The display light L ₁ which is visible light is projected. The display light L ₁ is reflected by the front windshield 2 to generate a virtual image V, and causes the driver D of the vehicle to visually recognize the virtual image V by superimposing it on the actual scene transmitted through the front windshield 2.

Further, the HUD ₁ has a DMS (driver monitoring system) function of monitoring the state of the driver D by a viewpoint detection system, projects infrared light L2 onto the driver D, captures the driver D, and captures an image. The position of the viewpoint of the driver D can be detected based on the above.

Specifically, the HUD 1 is covered with a housing 3 molded from a black ABS resin or the like to prevent the intrusion of external light and is partitioned from the outside, and the housing 3 is a transparent resin such as polycarbonate (not shown). An opening (transmissive portion) 4 covered with is formed. A display unit 5, a folding mirror 6, a concave mirror 7, an infrared light irradiation unit 8, a camera 9, and a control unit 10 are held and housed inside the housing 3.

The display unit 5 is provided with a light source and a liquid crystal panel made of a chip-type light emitting diode, and the liquid crystal panel _two -dimensionally modulates the emitted light of the light source to generate visible light (display light L1). Display by projection. The folding mirror 6 is formed by depositing a metal such as aluminum on a resin such as polycarbonate molded so as to have a flat portion, and simply reflects light. The concave mirror 7 is formed by depositing a dielectric multilayer film or the like on a resin such as polycarbonate molded so as to have a concave portion, and has a property of magnifying and reflecting visible light and transmitting infrared light.

The infrared light irradiation unit 8 is provided on the back side of the concave mirror 7 (opposite the opening 4 and the folding mirror 6 with respect to the concave mirror 7), and emits infrared light (near infrared rays) emitted by a light source composed of a light emitting diode. Irradiate toward. The camera 9 includes an image pickup element that is sensitive to infrared light in the wavelength band emitted from the infrared irradiation unit 8 and a lens that can transmit infrared light and form an image on the image pickup device, and captures a near-infrared image. Take a picture. In order to selectively transmit infrared light, a filter that absorbs or reflects wavelengths such as visible light may be used.

The control unit 10 is composed of a microprocessor, a memory, various electronic components for operating them, a board, and a case, and is a display unit so as to appropriately display the image of the HUD1 based on the vehicle information and the input information of the driver D. 5 is controlled.

Further, as shown in FIG. 2, the control unit 10 determines the position of the _iris (black eye) IL of the driver _D' s left eye EL and the right eye E based on the gradation difference (contrast) of the image P captured by the camera 9. The position of the iris _{IR of R is} detected, and the position of the midpoint _IC of the left and right _{irises IL} and IR is detected as the viewpoint position. The detected viewpoint position can be used for detecting the state of the driver D (looking aside, taking a nap, etc.), and is output to the outside as viewpoint information.

That is, the control unit 10 has the nose tip N, the _philtrum R, and the chin F as feature points for recognizing the approximate position of the left eye EL of the driver D based on the gradation difference of the image P captured by the camera 9. The left corner of the mouth _ML and the left eyelid _BL are detected, and the inner corner E _1L , the outer corner of the eye E _2L , and the upper eyelid E _3L are detected as characteristic points related to the left eye _EL , and the position of the iris _IL is detected.

Further, the control unit 10 has the nose tip N, the philtrum _R , and the chin F as feature points for recognizing the approximate position of the driver D's right eye ER based on the gradation difference of the image P captured by the camera 9. The right corner of the mouth MR and the right eyelid _BR are detected, and the inner corner E _1R , the outer corner of the eye E _2R , and the upper eyelid E _3R are detected as feature points related to the right eye _ER , and the position of the _{iris IR} is detected.

In the HUD 1, the display light L ₁ from the display unit 5 is reflected by the folding mirror 6, then magnified and reflected by the concave mirror 7, passes through the opening 4, and is projected onto the front windshield 2. The display light L ₁ projected on the front windshield 2 is enlarged and reflected toward the driver D, generates a virtual image V, and is displayed on the driver D over the actual scene transmitted through the front windshield 2.

On the other hand, the infrared light L ₂ from the infrared light irradiation unit 8 passes through the concave mirror 7, passes through the opening 4, is projected onto the front windshield 2, and is reflected by the front windshield 2 toward the driver D. And irradiate the driver D. Then, when reflected by the driver D, a part of the infrared light L ₂ follows the reverse path, and the driver D is imaged by the infrared light L ₂ transmitted through the concave mirror 7 and incident on the camera 9. The captured image P is input to the control unit 10. This imaging is performed periodically or irregularly during the display of the virtual image V, and here, a moving image at a periodic frame rate is performed while the virtual image V is displayed.

When the captured image P of the driver D is input, the control unit 10 detects the positions of the left and right irises IL and IR of the driver _D as described above, and the intermediate point I of the _{iris IL} and _IR . The position of _C is detected as the viewpoint position. However, originally, the captured image P of the driver D as shown in FIG. 2 should be obtained, but as shown in FIG. ₃ , the infrared component of the strong external light L3 such as sunlight is the front windshield. 2. When the image is transmitted to the camera 9 through the opening 4 and the concave mirror 7, an overexposed area S1 as shown in _FIG . ₄ is generated in the captured image P, and the gradation difference of the peripheral area S2 is increased. When the area _{S 2} contains feature points for detecting the left eye _EL and the right eye ER, it becomes difficult to detect the position of the feature points, and the iris _IL and the iris _IR (iris I in FIG. 4). _R ) becomes difficult to detect.

Therefore, when it is difficult for the control unit 10 to detect the position of the feature point and it is difficult to detect one position of the _{iris IL} and _IR , the control unit 10 takes only the other position of the iris _IL and IR in the captured image P. _Detects based on the gradation difference, and for one position of the iris _IL or IR, a gamma-corrected image with the gamma value of the captured image P adjusted is generated, and detection is performed based on the gradation difference in the gamma-corrected image. ..

For example, in the captured image P of FIG. 4, for the left eye _{EL, the nose tip N, the human middle R, the jaw F, the mouth angle ML, the inner corner of the eyelid BL, the inner corner of the eye E 1L , the outer} corner of the eye E _2L , and the upper eyelid E. All _3L can be detected and the position of the iris _IL can be detected, but for the right eye ER, the inner corner of the eye _BR , the inner corner of the eye E _1R , the outer corner of the eye E _2R and the upper eyelid E _3R are _detected based on the gradation difference. Since the position of the iris _IR cannot be detected because it cannot be detected, the control unit 10 has a feature point (eyelid _BR ,) in which the position of the captured image P is difficult to detect, as shown in FIG. The region S3 including the inner corner of the eye E _1R , the outer corner of the eye E _2R and the upper eyelid E _3R ) was determined _{(the region S 3 is} here the known feature points (nose tip N, human _R , jaw F, mouth angle MR, inner corner of the eyelid). The approximate position of the right eye _ER is estimated based on ( _1L of the inner corner of the eye, the outer corner of the eye E _2L , or the upper _eyelid E _3L ), and the position is detected as the peripheral region of the right eye _ER estimated from the position. It is determined to include difficult feature points) _, the gamma value of region S3 is adjusted to generate a gamma-corrected image P _γ . In the gamma-corrected image P _γ , the control unit 10 can detect the inner corner of the eye _BR , the inner corner of the eye E _1R , the outer corner of the eye E _2R , and the upper eyelid E _3R , which could not be detected in the captured image P. The position of _R , and thus the position of the viewpoint, can be detected.

The method of generating the gamma-corrected image P _γ is not particularly limited as long as the gradation difference can be sufficiently obtained, and the gamma value of only the region _S3 of the captured image P is not adjusted but is captured. The gamma value of the entire image P may be adjusted.

For example, the gamma value of the captured image P (the image sensor (CMOS, CCD, etc.) of the camera 9 generates a linear signal with respect to the input light amount, and the gamma value is applied to this signal (input) to convert the data (output). ) Is the gamma value when recorded as the captured image P) When γ is γ = α, as shown in FIG. 6 (α = 1 in FIG. 6), the input of the captured image P is from 0 to less than a predetermined threshold. The gamma-corrected image Pγ can be generated by adjusting γ to be larger than α in the range up to. In the gamma curve of FIG. 6, the middle ground 128 of the inputs from 0 to 255 is set as the threshold, the curve is convex downward with γ> α (= 1) in the range below the threshold, and γ = α (γ = α) in the range above the threshold. By setting the linear characteristic in 1), a gamma-corrected image P _γ is generated in which the gradation difference in the region where the brightness of the captured image P is too low to obtain a sufficient gradation difference is enlarged.

Also in FIG. 7, when the gamma value γ of the captured image P is γ = α (= 1), the gamma value is adjusted to be larger than α within the range where the input of the captured image P is less than the threshold value, and the gamma-corrected image is also obtained. An example of generating Pγ is shown. In the gamma curve of FIG. 7, the gamma curve has a turning point near the input of 80 in the range below the threshold with the threshold set to 176, and γ> α (=) in the range below the turning point. It has a downwardly convex curve in 1), an upwardly convex curve in the range above the turning point with γ> α (= 1), and a linear characteristic with γ = α (= 1) in the range above the threshold. As a result, when the brightness of the portion where the sufficient gradation difference is obtained is reduced according to the gamma curve of FIG. 6 and the gradation difference is reduced, the gamma-corrected image P that prevents such reduction of the gradation difference is prevented. _γ is generated.

FIG. 8 shows a gamma-corrected image in which when the gamma value γ of the captured image P is γ = α (= 1), the gamma value is adjusted to be smaller than α in the range where the input of the captured image P is equal to or greater than the threshold value. An example of generating Pγ is shown. Similar to the gamma curve of FIG. 6, the gamma curve of FIG. 8 has a threshold of 128 and is convex downward with γ> α (= 1) in the range below the threshold, but γ <α (=) in the range above the threshold. In 1), it is convex upward (strictly speaking, in FIG. 8, the threshold value is α = 1, which is a turning point, and γ <α holds in the range exceeding the threshold value), whereby the range above the threshold value is established. A gamma-corrected image P _γ is generated in which the brightness is higher than that of the captured image P and reinforces the effect of the gamma curve of FIG.

If the control unit 10 has difficulty in detecting the position of the feature point and it is difficult to detect the positions of both the left and right _irises IL and IR, the control unit 10 does not generate the gamma-corrected image P _γ and the left and right _iris I. After transitioning to a state in which one of the positions of _L and _IR can be detected, the gamma-corrected image P _γ is generated and the position of the feature point based on the gamma-corrected image P γ (and the position of the iris _IL and _IR and the viewpoint position) are detected. Start.

That is, the control unit 10 is not affected by strong light by turning to the front or returning to the front from the state where the driver D is facing sideways or tilting his / her posture and his / her face is out of the shooting range. Although the left and right _irises IL and IR can be detected, only at the moment when one of the _{irises IL} and _IR cannot be detected due to strong light, the _{iris IL} and IR can be detected. The other position is detected based on the captured image P, and one position of the _{iris IL} and IR is detected based on the gamma-corrected image P _γ , and the viewpoint position is detected. After that, when the influence of the strong light disappears and one of the _{iris IL} and IR can be detected, the control unit 10 stops the generation of the gamma-corrected image P _γ and both the _{iris IL} and IR are stopped. The position is detected based on the captured image P.

The viewpoint detection system according to the present embodiment is based on the gradation difference between the camera 9 that captures the driver D by the infrared light L ₂ projected on the driver D of the vehicle 2 and the captured image P of the camera 9. , The position of the feature point of the driver D is detected, and the positions of the left and right _irises IL and IR of the driver _D in the captured image P are detected based on the detected feature points, and the detected left and right irises. A _control unit 10 for _detecting the viewpoint position of the driver _D based on the positions of the IL and IR is provided, and the control unit 10 has difficulty in detecting the position of the feature point, and the left and right irises IL and _IR When it is difficult to detect one position, a gamma-corrected image P _γ with the gamma value of the captured image P adjusted is generated, and the feature points that were difficult to detect based on the gradation difference in the gamma-corrected image P _γ . Since the position is detected, even if the gradation difference in the captured image P is not sufficient, the viewpoint position of the driver D is detected by generating the gamma-corrected image P _γ with the adjusted gamma value and securing the gradation difference. can do.

In this embodiment, when it is difficult for the control unit 10 to detect the position of the feature point and it is difficult to detect the position of one of the left and right _{irises IL} and IR, it is difficult to detect the position of the captured image P. Since the gamma-corrected image P _γ is generated by adjusting the gamma value of the region S3 including the characteristic points (in FIG. ₄ , the inner corner of the eye _BR , the inner corner of the eye E _1R , the outer corner of the eye E _2R and the upper eyelid E _3R ), the captured image P The image processing area is limited and the processing time can be shortened as compared with adjusting the entire gamma value.

Further, when it is difficult to detect the position of the feature point and it is difficult to detect both the positions of the left and right _irises IL and IR, the control unit 10 does not generate the gamma-corrected image P _γ and the left and right _irises I. Since the generation of the gamma-corrected image P _γ and the detection of the position of the feature point based on the gamma-corrected image P γ are started after one of the positions of _L and _IR transitions to the detectable state, the adjustment of the gamma value is unnecessary or meaningless. By not generating the gamma-corrected image P _γ under the conditions, the delay in detecting the viewpoint position and the delay in outputting the viewpoint information to the outside due to the generation of the unnecessary or meaningless gamma-corrected image P _γ are suppressed. Can be done.

Although the embodiments for carrying out the present invention have been illustrated above, the embodiments of the present invention are not limited to those described above, and may be appropriately modified without departing from the spirit of the invention.

For example, in the above embodiment, the position of the intermediate point IC of the left and right _{irises IL} and _IR is set as the viewpoint position, but the position of the iris _IL and _IR itself and other positions may be set as the viewpoint position.

Further, a portion other than the portion shown in FIG. 2 may be detected as a feature point, and a portion other than the iris (for example, the position of the pupil) may be detected as the eye position.

Further, the method of setting the threshold value is also arbitrary, and a plurality of threshold values may be set. As an example, a first threshold value and a second threshold value larger than the first threshold value are set, and a region below the first threshold value ( At least, the gamma value is adjusted so that γ> α in the range (including the region where the brightness is too low to obtain the necessary and sufficient gradation difference), and the region exceeding the second threshold value (at least, the brightness). Gamma-corrected images may be generated by adjusting the gamma value so that γ <α in the range (including the region where is too high to obtain the necessary and sufficient gradation difference). The gamma curve of No. 7 corresponds to the case where only the first threshold value is set, and the gamma curve of FIG. 8 corresponds to the case where the first threshold value and the second threshold value match.)

2 Vehicle 9 Camera (imaging means)
10 Control unit (feature point detection means, viewpoint position detection means)
_BL eyebrows (feature points)
_BR eyebrows (feature points)
D Driver (user)
E _1L inner corner (feature point)
E _1R inner corner (feature point)
E _2L outer corner of the eye (feature point)
E _2R outer corner of the eye (feature point)
E _3L upper eyelid (feature point)
E _3R upper eyelid (feature point)
E _L Left eye E _R Right eye F Jaw (feature point)
_IL iris _IR iris _ML mouth corner (feature point)
_MR mouth angle (feature point)
N Nose tip (feature point)
P Captured image P _γ Gamma corrected image R Philtrum (feature point)

Claims (5)

An image pickup means for imaging the user by infrared light projected on the user of the vehicle, and an imaging means.
The feature point detecting means for detecting the position of the feature point of the user based on the gradation difference in the captured image of the imaging means, and the feature point detecting means.
Based on the feature points detected by the feature point detecting means, the positions of the left and right eyes of the user in the captured image are detected, and the viewpoint position of the user is detected based on the detected positions of the left and right eyes. Equipped with a viewpoint position detecting means to detect
The feature point detecting means adjusts the gamma value of the captured image when it is difficult to detect the position of the feature point and it is difficult for the viewpoint position detecting means to detect the position of one of the left and right eyes. A viewpoint position detection system characterized in that an image is generated and the position of the feature point is detected based on the gradation difference in the gamma-corrected image. When it is difficult for the feature point detecting means to detect the position of the feature point and it is difficult for the viewpoint position detecting means to detect the position of one of the left and right eyes, it is difficult to detect the position of the captured image. The viewpoint position detection system according to claim 1, wherein the gamma-corrected image is generated by adjusting the gamma value of a region including a feature point. When the gamma value of the captured image is α, the feature point detecting means adjusts the gamma value to be larger than α within a range where the input of the captured image is less than a predetermined threshold value to obtain the gamma-corrected image. The viewpoint position detection system according to claim 1 or 2, wherein the viewpoint position detection system is generated. When the gamma value of the captured image is α, the feature point detecting means adjusts the gamma value to be smaller than α in the range where the input of the captured image is equal to or greater than the threshold value to generate the gamma-corrected image. 3. The viewpoint position detection system according to claim 3. The feature point detecting means is characterized in that it does not generate the gamma-corrected image when it is difficult to detect the position of the feature point and it is difficult for the viewpoint position detecting means to detect the positions of both the left and right eyes. The viewpoint position detection system according to any one of claims 1 to 4.

Images