JP2016170052A - Eye detector and vehicle display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable the stable detection of a human eye regardless of the brightness of a subject.SOLUTION: An eye detector 200 includes: an imaging section 210 where multiple imaging elements 211, 212 different in exposure are arranged and each one of the multiple imaging elements 211, 212 image a subject by receiving imaging light N1, N2 from the subject; an image synthesis section 221 for synthesizing the images F, S of the subject captured by respective imaging elements 211, 212 and generating a high-dynamic range image G having a dynamic range larger than those of the images F, S of the subject captured by the respective imaging elements 211, 212; and an analysis section 222 for detecting a human eye from the high-dynamic range image G.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an eye detection device that detects a human eye from an image obtained by imaging a human face, and a vehicle display system using the eye detection device.

  2. Description of the Related Art Conventionally, as this type of eye detection apparatus, an apparatus that images a subject including a human face using an image sensor such as a CCD or a CMOS is known (for example, see Patent Document 1). Based on the eyes of the person detected by the eye detection device, the position and line-of-sight direction of the person's eyes can be further calculated. For example, as shown in Patent Document 2, a display image (virtual image) in a head-up display device can be calculated. It can be used for position adjustment.

JP-A-4-68500 JP 2008-126984 A

  However, if the subject is too bright, such as when the sunlight is strong and the subject is so bright, a so-called white-out phenomenon occurs where the originally bright and dark areas become a single color, and conversely, the subject does not receive light in a dark place such as a tunnel. When the subject is too dark, for example, a so-called black crushing phenomenon occurs in which the dark portion becomes a solid black color. In either case, there is a problem that the human eye cannot be detected. As a countermeasure against such a problem when the subject is dark, as disclosed in Patent Document 2, there can be considered a method of imaging the subject by irradiating the subject with infrared light using an infrared camera as an imaging element. However, even when an infrared camera is used, if the subject is too bright, a whiteout phenomenon occurs.

  The present invention has been made in view of the above problems, and an object thereof is to provide an eye detection device and a vehicle display system capable of stably detecting human eyes regardless of the brightness of a subject. To do.

In order to achieve the above object, an eye detection device of the present invention is an eye detection device that images a subject including a human face and detects a human eye based on the image of the captured subject.
An imaging unit that includes a plurality of imaging elements having different exposures, and each of the plurality of imaging elements receives imaging light from the subject, and images the subject;
Image synthesis for synthesizing images of the subject imaged by each of the plurality of image sensors and generating a high dynamic range image having a wider dynamic range than the image of the subject imaged by each of the plurality of image sensors. And
An analysis unit for detecting human eyes from the high dynamic range image.

  In order to achieve the above object, a display system for a vehicle according to the present invention calculates the position and / or line of sight of a user's eyes by the above-described eye detection device, and calculates the user's eyes calculated by the eye detection device. According to the position and / or line of sight, the position of the virtual image of the display image viewed by the user is adjusted by display light indicating the display image emitted from the head-up display device.

  According to the present invention, it is possible to stably detect human eyes regardless of the brightness of the subject.

It is a figure explaining the composition of the display system for vehicles in the embodiment of the present invention. It is a figure explaining the scenery which the user of the vehicle in the said embodiment visually recognizes. It is a general-view figure which shows the eye detection apparatus in the said embodiment. It is a figure which shows the 1st, 2nd to-be-photographed image and synthetic | combination image in the said embodiment. It is a flowchart figure explaining operation | movement of the display system for vehicles in the said embodiment.

FIG. 1 shows a system configuration of a vehicle display system 1 according to this embodiment to which the eye detection device of the present invention is applied.
The vehicle display system 1 according to the present embodiment projects display light L representing a display image onto a windshield (one embodiment of a reflection / transmission surface) 2a of the vehicle 2, and a virtual image M of the display image is displayed on the occupant ( A head-up display device (hereinafter referred to as a HUD device) 100 that is visually recognized by the user 3 who is a driver) and a subject including the face of the user 3 are imaged, and the eyes of the user 3 are detected to detect the eyes. An eye detection device 200 that calculates the position and line of sight, a navigation system 300 that generates information for route guidance, and a display that controls display of the HUD 100 based on information input from the eye detection device 200 and the navigation system 300 And a controller 400.

  The HUD device 100 includes a display 10 that displays a display image J for displaying a virtual image M on a display surface, a flat mirror 20 that reflects image light K indicating the display image J, and an image reflected by the flat mirror 20. A free-form surface mirror 30 that expands and deforms the light K and reflects it as the display light L toward the windshield 2a.

  The display 10 displays a route guidance image indicating a guidance route, a vehicle information image indicating information on the vehicle 2, and the like on the display surface as a display image J under the control of the display controller 400 described later. It comprises a transmissive liquid crystal display composed of a display element (not shown) such as a liquid crystal panel and a light source (not shown) that illuminates the display element. The display 10 is not a transmissive liquid crystal display, but a self-luminous organic EL display, a reflective DMD (Digital Micromirror Device), a reflective and transmissive LCoS (registered trademark: Liquid Crystal on Silicon), or the like. It may be constituted by. The display controller 400 to be described later displays on the display surface of the display 10 according to the position and line-of-sight direction of the user 3 calculated by the eye detection device 200 so that the user 3 visually recognizes the virtual image M. Adjust the display position of image J. Thereby, the user 3 can visually recognize the virtual image M aligned with the tip of the line of sight. As another method for adjusting the display position of the virtual image M, for example, an angle adjustment unit including a motor for adjusting the angle of the free-form surface mirror 30 may be provided.

  The plane mirror 20 reflects the image light K emitted from the display 10 toward the free-form curved mirror 30.

  The free-form surface mirror 30 is formed by forming a reflective film on the surface of a concave base material made of, for example, a synthetic resin material by means such as vapor deposition, and enlarges the image light K reflected by the flat mirror 20 and K is transformed and emitted as display light L in the direction of the windshield 2a.

  FIG. 2 shows an example of the virtual image M visually recognized by the user 3. In the example shown in FIG. 2, the virtual image M that is visually recognized on the back side of the windshield 2a when viewed from the user 3 has a route guidance virtual image M1 that guides the route to the destination. The route guidance virtual image M1 is, for example, an image having a shape such as an arrow head or an arrow, and indicates a specific direction. The route guidance virtual image M1 may include a character image such as “turn left 200m”.

  The above is the optical configuration of the HUD device 100 in the present embodiment, and the display light L emitted from the HUD device 100 is projected onto the windshield 2a of the vehicle 2 and enters the eyes of the user 3, The user 3 is made to visually recognize the virtual image M in a predetermined displayable area E of the windshield 2a above the steering wheel 2b. The displayable area E of the windshield 2a corresponds to the display area of the display device 10. By moving the display image J within the displayable area E of the display device 10, the displayable area E of the windshield 2a. The virtual image M moves and is visually recognized.

  The eye detection device 200 captures an image of a subject including the face portion of the user 3, detects the eye of the user 3 from the image of the subject, and calculates the position and line of sight of the user 3; In the embodiment, an imaging unit 210 that is disposed in front of the user 3 and images a subject, and an image analysis unit 220 that analyzes an image of the subject captured by the imaging unit 210 are provided.

  As shown in FIG. 3, the imaging unit 210 includes first and second imaging elements 211 and 212 and a transflective member 213. The first and second imaging elements 211 and 212 are made of, for example, a CCD or a CMOS, and are configured to have different exposures (amounts of sensitive light). The first and second imaging elements 211 and 212 are arranged at different angles, and each face the transflective member 213. The transflective unit 213 receives the imaging light N from the subject, transmits part of the imaging light N as the first imaging light N1 toward the first imaging element 211, and partially transmits the second imaging. The light N2 is reflected toward the second image sensor 212. It is assumed that the ratio between the transmissivity and the reflectivity of the transflective portion 213 is 1: 1. Accordingly, the first imaging light N1 and the second imaging light N2 are both the imaging light N in which the light amount is halved. As a result, the first and second imaging elements 211 and 212 receive the first and second imaging lights N1 and N2 to which the imaging light N is distributed by the transflective portion 213, respectively, and each of the subjects picks up the subject from the same angle. The first image sensor 211 can transmit the first subject image F and the second image sensor 212 can transmit the second subject image S to the image analysis unit 210 at the same timing. FIG. 4A shows an example of the first subject image F, and FIG. 4B shows an example of the second subject image S. The first and second subject images F and S are images with different exposures. In the example shown in FIG. 4, the first subject image F is exposed more than the second subject image S.

  The image analysis unit 220 includes an image synthesis unit 221 and an analysis unit 222. The image composition unit 221 synthesizes the first subject image F transmitted from the first image sensor 211 and the second subject image S transmitted from the second image sensor 212 to produce a composite image G (high dynamic range). Image). The analysis unit 222 detects the eyes of the user 3 from the composite image G generated by the image composition unit 221, calculates the eye position and line of sight of the user 3, and displays the eye position information I1 and line-of-sight information I2 This is transmitted to the controller 400.

  Next, with reference to FIG. 5, processing from the detection of the position of the eyes of the user 3 by the image analysis unit 220 to the position adjustment of the virtual image M by the display controller 400 in the vehicle display system 1 will be described.

  First, in step S101, the image analysis unit 220 receives the first and second subject images F and S from the first and second imaging elements 211 and 212.

  Next, in step S102, the image composition unit 221 initializes the luminance total value Bt of the composite image G, which is an internal variable.

  Next, in step S103, the image composition unit 221 performs composition of the pixels of the first subject image F and the pixels of the second subject image S (composites pixels). Specifically, a value obtained by multiplying the pixel brightness value Bf of the first subject image F by the first exposure coefficient Cf and a value obtained by multiplying the pixel brightness value Bs of the second subject image S by the second exposure coefficient Cs. Are added to obtain the pixel luminance value Bg of the composite image G (Bg = Bf × Cf + Bs × Cs). Therefore, the pixel luminance value Bg of the composite image G can take a wider range than the possible range of the pixel luminance value Bf of the first subject image F and the pixel luminance value Bs of the second subject image S. The first and second exposure coefficients Cf and Cs are set in advance according to the exposure times of the first and second imaging elements 211 and 212 and the lens aperture.

  Next, in step S104, the image composition unit 221 calculates the total luminance value Bt of the composite image G. Specifically, the pixel luminance value Bg of the composite image G calculated in step S103 is added to the luminance total value Bt (Bt = Bt + Bg).

  Next, in step S105, the image composition unit 221 determines whether or not the processes in steps S103 and S104 have been completed for all the pixels in the first horizontal line of the first and second subject images F and S. . If it is determined that the processing in steps S103 and S104 has been completed for all pixels in one horizontal line (step S105; Yes), the pixel composition unit 221 proceeds to step S106, and has not ended. Is determined (step S105; No), the process returns to step S103, and the processes of steps S103 and S104 are repeatedly executed for the next pixel in the horizontal line.

  Next, in step S106, the image composition unit 221 determines whether or not the processing in steps S103 and S104 has been completed for all the horizontal lines of the first and second subject images F and S. When it is determined that the processing in steps S103 and S104 has been completed for all horizontal lines (step S106; Yes), the image composition unit 221 proceeds to step S107 and determines that the processing has not been completed. (Step S106; No), the process returns to Step S103, and the processes of Steps S103 and S104 are repeatedly executed for the pixels of the next horizontal line. In this way, the processes in steps S103 and S104 are performed on all the pixels of the first and second subject images F and S. The synthesized image G composed of the pixels synthesized in this way becomes a high dynamic range image having a wider dynamic range than the first and second subject images F and S. FIG. 4C shows an example of the composite image G. As shown in FIG. 4C, an image in which whiteout and blackout are reduced with respect to the first and second subject images F and S is obtained as the composite image G.

  Next, in step S107, the image composition unit 221 calculates the average brightness value Ba of the composite image G. Specifically, the total luminance value Bt of the composite image G calculated in step S104 is divided by the number of pixels d of the composite image G (Ba = Bt / d).

  Next, in step S108, the image composition unit 221 stores the composite image G obtained by the processing up to step S107 and the luminance average value Ba of the composite image G.

  Next, in step S <b> 109, the analysis unit 222 detects the eyes of the user 3 from the composite image G stored in the image composition unit 221 using the average brightness value Ba of the composite image G. Specifically, a predetermined luminance range (luminance range) is set based on the average luminance value Ba of the composite image G, and edge detection is performed on a portion (pixel) within the set luminance range of the composite image G. To detect eye contours and black eyes (that is, edge detection is not performed on extremely high or low luminance portions). As described above, the composite image G is an image in which overexposure and underexposure are reduced with respect to the first and second subject images F and S, so that the user can stably operate regardless of the brightness of the subject. The third eye can be detected. Further, since the composite image G has a wide dynamic range, it may be considered that it takes more time to detect the eyes than before when edge detection is performed on all pixels, but the average luminance value Ba of the composite image G is used. By narrowing down the part where edge detection is performed, it is possible to detect eyes quickly.

  Next, in step S110, the analysis unit 222 calculates the eye position and line of sight of the user 3 from the correlation between the eye contour and the black eye detected in step S109.

  Next, in step S111, the analysis unit 222 transmits the position information I1 and line-of-sight information I2 of the user 3 calculated in step S110 to the display controller 400.

  Next, in step S112, the display controller 400 calculates the correction amount of the display position of the display image J based on the eye position information I1 and the line-of-sight direction information I2 received from the eye detection device 200. Specifically, the eye position information I1 and the line-of-sight information I2 received from the eye detection device 200 are based on a correction data table in which display correction amounts for the eye position and the line-of-sight are stored in a storage unit (not shown) in advance. Read the display correction amount.

  Next, in step S113, the display controller 400 updates the display image J based on the display correction amount calculated in step S112 (moves the display position), and the virtual image M is overlapped with the line of sight of the user 3. Adjust the position.

  Each unit of the vehicle display system 1 repeatedly executes the above processing.

As described above, the eye detection apparatus 200 according to the present embodiment captures an image of a subject including the face of a person (user 3), and detects the eyes of the person based on the captured image of the object. The imaging unit 210 includes a plurality of imaging elements 211 and 212 having different exposures, and each of the imaging elements 211 and 212 receives imaging light N1 and N2 from the subject and images the subject. The image F, S of the subject imaged by each of the image pickup devices 211, 212 is synthesized and has a wider dynamic range than the image F, S of the subject imaged by each of the plurality of image pickup devices 211, 212. An image composition unit 221 that generates a high dynamic range image (composite image G) and an analysis unit 222 that detects human eyes from the high dynamic range image are provided.
According to this, it is possible to stably detect human eyes regardless of the brightness of the subject.

The eye detection device 200 receives the imaging light N from the subject, transmits a part of the imaging light N from the subject, reflects a part thereof, and directs the imaging light N toward each of the plurality of imaging elements 211 and 212. An outgoing transflective member 213 is provided.
According to this, a plurality of subject images F and S having different exposures can be simultaneously captured by a plurality of imaging elements 211 and 212 and transmitted to the image composition unit 221 at the same timing, and a high dynamic range image is generated. The time required to do so can be shortened.

In the eye detection apparatus 200, the image composition unit 221 calculates the average value Ba of the luminance total value Bt of the high dynamic range image, and the analysis unit 222 performs a predetermined operation based on the average value Ba of the luminance total value Bt. A luminance range is set, and human eyes are detected from a portion within the set luminance range of the high dynamic range image.
According to this, even when a high dynamic range image is used, it is possible to quickly detect human eyes.

The vehicle display system 1 according to the present embodiment calculates the position and / or line of sight of the eye of the user 3 by the eye detection device 200, and the position and / or line of sight of the eye of the user 3 detected by the eye detection device 200. Accordingly, the position of the virtual image M of the display image J visually recognized by the user 3 is adjusted by the display light L indicating the display image J emitted by the head-up display device 100.
According to this, the position of the virtual image M can be adjusted stably regardless of the surrounding brightness.

  Note that the present invention is not limited to the above-described embodiment, and various modifications (including addition and deletion of components) can be made without departing from the scope of the invention. For example, the eye detection device 200 and the display controller 400 may be integrally configured in one control unit. Further, the eye detection device 200 may determine not only the position and line of sight of the eye but also the falling asleep or looking aside, a decrease in concentration, etc. based on the detected eye of the person. Further, the number of image sensors is not limited, and may be three or more. Furthermore, a plurality of transflective members may be provided in accordance with the number of image sensors.

  INDUSTRIAL APPLICABILITY The present invention is suitable for an eye detection device that detects a human eye from an image obtained by capturing a human face and calculates an eye position and / or line of sight and a vehicle display system using the eye detection device.

DESCRIPTION OF SYMBOLS 1 Display system for vehicles 2 Vehicle 3 User 100 Head up display apparatus 200 Eye detection apparatus 210 Image pick-up part 211 1st image pick-up element 212 2nd image pick-up element 220 Image analysis part 221 Image composition part 222 Analysis part 300 Navigation system 400 Display Controller F First subject image G Composite image (high dynamic range image)
K image light L display light M virtual image N imaging light N1 first imaging light N2 second imaging light S second subject image

Claims (4)

An eye detection device that images a subject including a human face and detects a human eye based on an image of the captured subject,
An imaging unit that includes a plurality of imaging elements having different exposures, and each of the plurality of imaging elements receives imaging light from the subject, and images the subject;
Image synthesis for synthesizing images of the subject imaged by each of the plurality of image sensors and generating a high dynamic range image having a wider dynamic range than the image of the subject imaged by each of the plurality of image sensors. And
An eye detection apparatus comprising: an analysis unit that detects a human eye from the high dynamic range image.   Comprising a transflective member that receives imaging light from the subject, transmits part of the imaging light from the subject, reflects part of the imaging light, and emits the reflected light toward each of the plurality of imaging elements; The eye detection device according to claim 1.   The image synthesizing unit calculates an average value of the total luminance value of the high dynamic range image, and the analysis unit sets a predetermined luminance range based on the average value of the total luminance value, and the high dynamic range image The eye detection apparatus according to claim 1, wherein a human eye is detected from a portion within a set luminance range.   The position and / or line of sight of the user's eyes calculated by the eye detection apparatus according to any one of claims 1 to 3 and detected by the eye detection apparatus. The display system for vehicles according to claim 1, wherein the position of the virtual image of the display image visually recognized by the user is adjusted by display light indicating the display image emitted by the head-up display device.

Images