JP2022190724A - Display control device, head-up display device and display control method - Google Patents

Abstract

To provide a display control device, a head-up display device and a display control method which give a motion parallax that does not give a sense of discomfort to an observer.SOLUTION: A display control device comprises: first image correction processing of acquiring the eye position and/or head position of a user in the vertical direction of the vehicle and the eye position and/or head position in the left-right direction of the vehicle, and correcting the position of an image to be displayed on a display unit with a first correction amount based on at least a change in the eye position or the head position in the vertical direction and in the eye position or the head position in the left-right direction; and second image correction processing of correcting the position of the image to be displayed on the display unit with a second correction amount obtained by multiplying the first correction amount by a first prescribed coefficient equal to or less than 1. The display control device executes the first image correction processing when the distance from the vehicle to a real object in the foreground is equal to or less than a first distance threshold, and executes the second image correction processing when the distance from the vehicle to the real object in the foreground is greater than the first distance threshold.SELECTED DRAWING: Figure 13

Description

The present disclosure relates to a display control device, a head-up display device, and a head-up display device that are used in a mobile object such as a vehicle and superimpose an image on the foreground of the mobile object (actual view in the forward direction of the mobile object seen from the vehicle occupant). It relates to a display control method and the like.

In Patent Document 1, display light projected onto a projection target portion such as a front windshield of a vehicle is reflected toward an occupant (observer) of the vehicle inside the vehicle, so that the observer can see the vehicle. describes a head-up display device (an example of a virtual image display device) that allows a user to visually recognize a virtual image that overlaps the foreground of the image. In particular, the head-up display device described in Patent Literature 1 virtually displays a display object at a predetermined position (here, the position is referred to as a target position) in the depth and vertical and horizontal directions in the real space of the foreground. (virtual image), and even if there is a change in the posture of the vehicle or the position of the observer's eyes, the display object will appear as if it were present at the target position in the foreground. Controls the image displayed inside the up display device. That is, such a head-up display device forms augmented reality in which virtual objects are added to the real scenery (foreground) and displayed, and changes in the attitude of the vehicle (this also changes the eye position of the observer with respect to the real scenery). connected) or when the position of the observer's eyes changes in the vehicle, the image displayed inside the head-up display device is changed according to the change in the observer's eye position detected by an eye position detection unit such as a camera. By correcting the position or the like, a motion parallax is given to the virtual object, and the observer can perceive the virtual object as if it were at the target position in the foreground (in the real scene).

JP 2010-156608 A

By the way, an eye position detector such as a camera detects the positions of the observer's eyes (right and left eye positions) by using a complex algorithm from captured images. Even if the object moves by only the amount of movement, it may be detected (erroneously detected) that the amount of movement exceeds a predetermined amount of movement due to detection errors or false detections depending on the detection environment. Then, the motion parallax image to the virtual object can be corrected according to the erroneously detected amount of movement. As a result, the position correction of the image in the left-right direction is greater than the amount of movement of the position of the observer's eyes, and the amount of movement is greater than the observer's real feeling, which may give the observer a sense of discomfort.

A summary of certain embodiments disclosed herein follows. It should be understood that these aspects are presented only to provide the reader with an overview of these particular embodiments and are not intended to limit the scope of this disclosure. Indeed, the present disclosure may encompass various aspects not described below.

An overview of the present disclosure relates to providing a display control device, a head-up display device, a display control method, and the like that provide a motion parallax that does not give an observer a sense of discomfort.

Therefore, the display control device, the head-up display device, the display control method, and the like described in this specification employ the following means in order to solve the above problems. This embodiment comprises at least a display (40) for displaying an image, and a relay optical system (80) for projecting the light of the image displayed by the display (40) onto a projection target member. A display control device (30) for performing display control in a head-up display device (20) for visually superimposing a virtual image of the image on the foreground, comprising: one or more processors (33); a memory (37); , one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33); and detecting real objects present in the vicinity of said vehicle. and a sensor (411) outside the vehicle, wherein the processor (33) detects the user's eye position and/or head position (Py) in the vertical direction of the vehicle and the eye position and/or head position (Py) in the horizontal direction of the vehicle. A position (Px) is obtained, and a first correction amount (Cx, Cy) a first image correction process (S160) for correcting the position of the image displayed on the display (40); a second image correction process (S170) for correcting with a second correction amount (Cxq1, Cyr1) obtained by multiplying (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is 1 or less; , performing the first image correction process when a distance (D4) from the vehicle to the real object in the foreground is less than or equal to a first distance threshold; The second image correction process (S170) is performed when the distance (D4) to the image is larger than the first distance threshold.

FIG. 1 is a diagram showing an application example of a vehicle virtual image display system to a vehicle. FIG. 2 is a diagram showing the configuration of the head-up display device. FIG. 3 is a diagram showing an example of a foreground visually recognized by an observer and a virtual image superimposed on the foreground while the host vehicle is running. FIG. 4 shows, in an embodiment in which the HUD device is a 3D-HUD device, a left-viewpoint virtual image and a right-viewpoint virtual image displayed on a virtual image plane, and a perceptual image perceived by an observer based on these left-viewpoint virtual images and right-viewpoint virtual images. , and is a diagram conceptually showing the positional relationship between . FIG. 5 is a diagram conceptually showing a virtual object placed at a target position in the real scene and an image displayed in the virtual image display area such that the virtual object is visually recognized at the target position in the real scene. FIG. 6 is a diagram for explaining a method of motion parallax addition processing in this embodiment. FIG. 7A is a comparative example showing a virtual image visually recognized from the position Px12 shown in FIG. 6 when the motion parallax adding process of this embodiment is not performed. FIG. 7B is a diagram showing a virtual image visually recognized from the position Px12 shown in FIG. 6 when the motion parallax adding process of this embodiment is performed. FIG. 8 is a diagram for explaining a method of motion parallax addition processing by moving the eye position (head position) in the vertical direction in this embodiment. FIG. 9 is a diagram showing an example of a foreground visually recognized by an observer and a virtual image displayed superimposed on the foreground while the own vehicle is running. FIG. 10 is a block diagram of a vehicle virtual image display system according to some embodiments. FIG. 11A is a flow diagram illustrating a method S100 for performing image correction operations based on observer eye position. FIG. 11B is a diagram explaining part of the flow diagram shown in FIG. 11A. FIG. 11C is a diagram explaining part of the flow diagram shown in FIG. 11A. FIG. 12 is an image diagram showing the eye position, the amount of change in the eye position, the movement speed of the eye position, etc. detected at each predetermined cycle time. FIG. 13 is a diagram for explaining the relationship between the ratio of the amount of correction to the amount of change in the eye position in the left-right direction and the up-down direction and the object distance.

1-6 and 7B-13, below, provide a description of the configuration and operation of an exemplary vehicular display system. In addition, the present invention is not limited by the following embodiments (including the contents of the drawings). Of course, modifications (including deletion of constituent elements) can be added to the following embodiments. In addition, in the following description, descriptions of known technical matters are omitted as appropriate in order to facilitate understanding of the present invention.

Please refer to FIG. FIG. 1 is a diagram showing an example of the configuration of a vehicle virtual image display system including a parallax 3D-HUD device. In FIG. 1, the left-right direction of a vehicle (an example of a moving body) 1 (in other words, the width direction of the vehicle 1) is the X axis (the positive direction of the X axis is the forward direction of the vehicle 1). leftward), and the vertical direction (in other words, the height direction of the vehicle 1) along a line segment that is orthogonal to the horizontal direction and orthogonal to the ground or a surface equivalent to the ground (here, the road surface 6) is the Y axis. (The positive direction of the Y-axis is the upward direction.), and the front-rear direction along a line segment perpendicular to each of the left-right direction and the vertical direction is the Z-axis (the positive direction of the Z-axis is the straight traveling direction of the vehicle 1.). do. This point also applies to other drawings.

As illustrated, the vehicle display system 10 provided in the vehicle (self-vehicle) 1 displays the positions of the left eye 700L and the right eye 700R of an observer (typically, the driver sitting in the driver's seat of the self-vehicle 1) and the line-of-sight direction. An eye position detection unit 409 for detecting a pupil (or face) for detecting a vehicle exterior sensor 411 configured by a camera (for example, a stereo camera) for imaging the front (broadly speaking, the surroundings) of the own vehicle 1, a head-up display device (hereinafter also referred to as a HUD device) 20, a display control device 30 that controls the HUD device 20, and the like.

FIG. 2 is a diagram showing one aspect of the configuration of the head-up display device. The HUD device 20 is arranged, for example, in a dashboard (reference numeral 5 in FIG. 1). The HUD device 20 accommodates a stereoscopic display device (an example of a display device) 40, a relay optical system 80, and the stereoscopic display device 40 and the relay optical system 80, and the display light K from the stereoscopic display device 40 is received from the inside. It has a housing 22 having a light exit window 21 capable of emitting light to the outside.

The stereoscopic display device 40 is assumed here to be a parallax 3D display device. This stereoscopic display device (parallax type 3D display device) 40 is an optical modulation element that is a naked-eye stereoscopic display device using a multi-viewpoint image display system capable of controlling depth expression by visually recognizing a left-viewpoint image and a right-viewpoint image. 50 and a light source unit 60 that functions as a backlight.

The light modulation element 50 has a light modulation element 51 that modulates the illumination light from the light source unit 60 to generate an image, and, for example, a lenticular lens or a parallax barrier (parallax barrier). The emitted light is divided into left-eye display light such as light rays K11, K12 and K13 (reference symbol K10 in FIG. 1) and right-eye display light such as light rays K21, K22 and K23 (FIG. 1). symbol K20) and an optical layer (an example of a light beam separating section) 52. The optical layer 52 includes optical filters such as lenticular lenses, parallax barriers, lens arrays, and microlens arrays. However, this is an example and is not limited. Embodiments of the optical layer 52 are not limited to the above-described optical filters, and may be the display light for the left eye (symbol K10 in FIG. 1) and the display light for the right eye (symbol K10 in FIG. 1) from the light emitted from the light modulation element 51. K20) includes all forms of optical layers placed in front or behind the light modulation element 51 . Some embodiments of the optical layer 52 are electrically controlled so that left-eye display light (K10 in FIG. 1) and right-eye display light (K10 in FIG. 1) are separated from light emitted from the light modulation element 51. symbol K20), such as a liquid crystal lens. That is, embodiments of optical layer 52 may include those that are electrically controlled and those that are not.

Also, in the stereoscopic display device 40, the light source unit 60 is configured with a directional backlight unit (an example of the light beam separating unit) instead of or in addition to the optical layer (an example of the light beam separating unit) 52. , left-eye display light such as light rays K11, K12, and K13 (reference symbol K10 in FIG. 1), and right-eye display light such as light rays K21, K22, and K23, etc. (reference symbol K20 in FIG. 1). , may be emitted. Specifically, for example, the display control device 30, which will be described later, causes the light modulation element 51 to display a left-viewpoint image when the directional backlight unit emits illumination light directed toward the left eye 700L. When the left-eye display light K10 such as K11, K12, and K13 is directed toward the viewer's left eye 700L, and the directional backlight unit emits illumination light toward the right eye 700R, the right viewpoint image is displayed on the light modulation element 51. By displaying, right-eye display light K20 such as light rays K21, K22, and K23 for the right eye is directed to the right eye 700R of the observer. However, this is an example and is not limited.

The display control device 30, which will be described later, performs, for example, image rendering processing (graphic processing), display device driving processing, and the like, so that the display light K10 for the left eye of the left viewpoint image V10 and the display light K10 for the left eye and the display light K10 for the right eye 700R of the observer's left eye 700L are displayed. By directing the right-eye display light K20 of the right-viewpoint image V20 and adjusting the left-viewpoint image V10 and the right-viewpoint image V20, the aspect of the perceptual virtual image FU displayed by the HUD device 20 (perceived by the observer) is controlled. be able to. The display control device 30, which will be described later, controls the display (the light modulation element 50) so as to generate a light field that (approximately) reproduces the light beams output in various directions from a point in a certain space. You may

The relay optical system 80 has curved mirrors (concave mirrors, etc.) 81 and 82 that reflect the light from the stereoscopic display device 40 and project the image display light K10 and K20 onto the windshield (projection target member) 2 . However, it may further include other optical members (refractive optical members such as lenses, diffractive optical members such as holograms, reflective optical members, or combinations thereof).

In FIG. 1, the stereoscopic display device 40 of the HUD device 20 displays images with parallax (parallax images) for the left and right eyes. Each parallax image is displayed as V10 and V20 formed on a virtual image display surface (virtual image formation surface) VS, as shown in FIG. The focus of each eye of the observer (person) is adjusted so as to match the position of the virtual image display area VS. Note that the position of the virtual image display area VS is referred to as an "adjustment position (or imaging position)", and is also referred to as a predetermined reference position (for example, the center 205 of the eyebox 200 of the HUD device 20, the observer's viewpoint position, the observation position). The distance from the person's head position or a specific position of the own vehicle 1 to the virtual image display area VS (see symbol D10 in FIG. 4) is referred to as an adjustment distance (imaging distance).

However, in reality, since the human brain fuses each image (virtual image), the human is at a position farther back than the adjustment position (for example, due to the convergence angle between the left viewpoint image V10 and the right viewpoint image V20). It is a fixed position, and the position perceived as being farther away from the observer as the angle of convergence decreases) is recognized as a perceptual image (here, an arrowhead figure for navigation) FU is displayed. do. Note that the perceptual virtual image FU may be referred to as a "stereoscopic virtual image", and may also be referred to as a "stereoscopic image" when the "image" is taken in a broad sense to include virtual images. It may also be referred to as a "stereoscopic image", "3D display", or the like. The HUD device 20 can display the left-viewpoint image V10 and the right-viewpoint image V20 so that the perceptual virtual image FU can be visually recognized at a position on the front side of the adjustment position.

Next, refer to FIGS. 3 and 4. FIG. FIG. 3 is a diagram showing an example of a foreground visually recognized by an observer and a perceptual image superimposed on the foreground and displayed while the own vehicle 1 is running. FIG. 4 is a diagram conceptually showing the positional relationship between the left-viewpoint virtual image and the right-viewpoint virtual image displayed on the virtual image plane, and the perceived image perceived by the observer from the left-viewpoint virtual image and the right-viewpoint virtual image. be.

In FIG. 3 , the own vehicle 1 is traveling on a straight road (road surface) 6 . The HUD device 20 is arranged inside the dashboard 5 . Display light K (K10, K20) is projected from the light exit window 21 of the HUD device 20 onto the projected portion (front windshield of the own vehicle 1) 2. FIG. In the example of FIG. 3, a first content image FU1 that is superimposed on the road surface 6 and indicates the route of the own vehicle 1 (indicating straight ahead here), and similarly the route of the own vehicle 1 (indicating straight ahead here). to display a second content image FU2 perceived farther than the first content image FU1.

As shown in the left diagram of FIG. 4, the HUD device 20 (1) detects the left eye 700L detected by the eye position detection unit 409, and at such a position and angle that the projection target 2 reflects the left eye 700L. to form a first left-viewpoint content image V11 at a predetermined position in the virtual image display area VS seen from the left eye 700L, and (2) reflected by the projection target unit 2 to the right eye 700R. Right-eye display light K20 is emitted to the projection target part 2 at a position and angle such that a first right-viewpoint content image V21 is formed at a predetermined position of the virtual image display area VS seen from the right eye 700R. do. The first content image FU1 perceived by the first left-viewpoint content image V11 and the first right-viewpoint content image V21 having parallax is positioned behind the virtual image display area VS by a distance D21 (the above position separated from the reference position by a distance D31).

Similarly, as shown in the right diagram of FIG. 4 , the HUD device 20 (1) detects the left eye 700L detected by the eye position detection unit 409 at such a position and angle as to be reflected by the projection target unit 2 . Left-eye display light K10 is emitted to the projection unit 2 to form a second left-viewpoint content image V12 at a predetermined position in the virtual image display area VS seen from the left eye 700L, and (2) the projected unit to the right eye 700R. 2, the display light K20 for the right eye is emitted to the projection target 2 at a position and angle such that it is reflected by the second right-viewpoint content image V22 at a predetermined position of the virtual image display area VS seen from the right eye 700R. to form an image. The second content image FU2 perceived by the second left-viewpoint content image V12 and the second right-viewpoint content image V22 having parallax is positioned behind the virtual image display area VS by a distance D22 (the above position separated from the reference position by a distance D32).

Specifically, the distance (imaging distance (first distance threshold) D10) from the reference position to the virtual image display area VS is set to a distance of, for example, "4 m". The distance (first perceptual distance D31) to the first content image FU1 shown in the left diagram is set to a distance of, for example, "7 m". to the content image FU2 (second perceptual distance D32) is set to, for example, "10 m". However, this is an example and is not limited.

FIG. 5 is a diagram conceptually showing a virtual object placed at a target position in the real scene and an image displayed in the virtual image display area such that the virtual object is visually recognized at the target position in the real scene. Note that the HUD device 20 shown in FIG. 5 shows an example of performing 2D display instead of 3D display. That is, the display device 40 of the HUD device 20 shown in FIG. 5 is a 2D display device that is not a stereoscopic display device (even a stereoscopic display device can perform 2D display). As shown in FIG. 5, when viewed from the viewer's eye position 700, the depth direction is the Z-axis direction, the left-right direction (the width direction of the vehicle 1) is the X-axis direction, and the vertical direction (the vehicle 1's vertical direction) is the Y-axis direction. Note that the direction away from the viewer is the positive direction of the Z axis, the leftward direction is the positive direction of the X axis, and the upward direction is the positive direction of the Y axis. direction.

The viewer's eye position 700 is a virtual object FU at a predetermined target position PT in the real scene by visually recognizing the virtual image V formed (imaged) in the virtual image display area VS through the projection target unit 2. perceive. A viewer visually recognizes the virtual image V of the image of the display light K reflected by the projection target section 2 . At this time, if the virtual image V is, for example, an arrow indicating a course, the arrow of the virtual image V is positioned in the virtual image display area VS so that the virtual object FU is placed at a predetermined target position PT in the foreground of the host vehicle 1 and visually recognized. to be displayed. Specifically, the HUD device 20 (display control device 30) uses the center of the observer's left eye 700L and right eye 700R as the origin of the projective transformation, and the virtual object FU of a predetermined size and shape arranged at the target position PT. An image to be displayed on the display 40 is rendered such that the virtual image V of a predetermined size and shape that has undergone the projective transformation is displayed in the virtual image display area VS. Then, the HUD device 20 (display control device 30) sets the virtual image display area VS so that the virtual object FU is perceived at the same target position PT as before the eye position is moved even when the eye position of the observer is moved. By changing the position of the virtual image V displayed in the target position PT, the virtual object FU (virtual image V) is displayed at the target position PT even though it is displayed at a position (virtual image display area VS) away from the target position PT. It can be recognized as if In other words, the HUD device 20 (display control device 30) changes the position of the image (virtual image V in the virtual image display area VS) on the display 40 based on the movement of the eye position based on the natural motion parallax. (In other words, the HUD device 20 adds motion parallax to the virtual image (image) by image correction accompanying the movement of the eye position. , making it easier to perceive depth).

In the description of the present embodiment, image position correction that expresses motion parallax in accordance with such a change in eye position is referred to as motion parallax addition processing (an example of eye-tracking image correction processing). The motion parallax adding process is not limited to image position correction that perfectly reproduces natural motion parallax, but may also include image position correction that approximates natural motion parallax. Note that the HUD device 20 (display control device 30) performs motion parallax addition processing (an example of eye-tracking image correction processing) in response to a change in the eye position 700. Based on the head position 710, motion parallax addition processing (an example of eye-tracking image correction processing) may be executed.

FIG. 6 is a diagram for explaining a method of motion parallax addition processing in this embodiment. The display control device 30 (processor 33) of the present embodiment controls the HUD device 20 to display the virtual images V41, V42, and V43 formed (imaged) in the virtual image display area VS via the projection target section 2. indicate. The virtual image V41 is set at the target position PT11, which is the perceived distance D33 (the position behind the virtual image display area VS by the distance D23), and the virtual image V42 is set at the perceived distance D34 (virtual image display The virtual image V43 is set at the target position PT12, which is a position behind the area VS by a distance D24 (>D23)), and the virtual image V43 is positioned at a perceived distance D35 longer than the perceived distance D34 of the virtual image V42 (more than the virtual image display area VS). The target position PT13 is set at a position on the far side by a distance D25 (>D24). Note that the amount of correction of the image on the display 40 corresponds to the amount of correction of the virtual image in the virtual image display area VS, so in FIG. The same codes C1, C2, and C3 are also used for the virtual image correction amounts (the same applies to codes Cy11 (Cy) and Cy21 (Cy) in FIG. 8).

When the observer's head position 710 (eye position 700) moves rightward (in the negative direction of the X axis) by ΔPx10 from the position indicated by reference sign Px11, the display control device 30 (processor 33) executes motion parallax addition processing. , the display positions of the virtual images V41, V42, and V43 displayed in the virtual image display area VS are corrected by the correction amounts C1 and C2, respectively, in the same direction as the observer's head position 710 (eye position 700). (>C1) and C3 (>C2) are corrected. FIG. 7A is a comparative example showing virtual images V41, V42, and V43 viewed from position Px12 shown in FIG. 7 is a diagram showing virtual images V44, V45, and V46 viewed from a position Px12 shown in FIG. 6 when parallax addition processing is performed; FIG. Note that in FIG. 7B, the difference in the positions of the virtual images V44, V45, and V46 is exaggerated so that the difference in correction amount can be easily understood. That is, the display control device 30 (processor 33) corrects the positions of the plurality of virtual images V41, V42, V43 according to the movement of the eye position due to the difference in the perceived distances D33, D34, D35 of the plurality of virtual images V41, V42, V43. By varying the amounts, the observer can perceive the motion parallax only between the virtual images V41 (V44), V42 (V45), and V43 (V46). More specifically, the display control device 30 (processor 33) increases the amount of correction in the motion parallax adding process as the perceived distance D30 set is longer, so that the plurality of virtual images V41 (V44) and V42 ( V45) and V43 (V46) are added with motion parallax.

FIG. 8 is a diagram for explaining a method of motion parallax addition processing when the eye position (head position) moves in the vertical direction in this embodiment. When the observer's head position 710 (eye position 700) moves upward (in the positive Y-axis direction) from the position indicated by Py12, the display control device 30 (processor 33) executes motion parallax adding processing to As shown in FIG. 8A, the position where the virtual image V displayed in the virtual image display area VS is displayed is the same direction (upward (Y-axis In the positive direction)), the correction amount Cy11 is corrected (the position of the virtual image V is changed from the position of the reference V48 to the reference V47). Further, when the observer's head position 710 (eye position 700) moves downward (Y-axis negative direction) from the position indicated by Py12, the display control device 30 (processor 33) executes motion parallax addition processing. As a result, the virtual image V displayed in the virtual image display area VS is displayed in the same direction (downward) as the observer's head position 710 (eye position 700) is moved, as shown in FIG. side (Y-axis negative direction)) by the correction amount Cy21 (the position of the virtual image V is changed from the position of V48 to V49). As a result, although the virtual object FU (virtual image V) is displayed at a position (virtual image display area VS) distant from the target position PT, it can be recognized as if it were at the target position PT ( It is possible to enhance the feeling as if the virtual object FU (virtual image V) is at the target position PT).

FIG. 9 is a diagram showing a real object 300 existing in the foreground and a virtual image V displayed by the HUD device 20 of the present embodiment, which is visually recognized when an observer faces forward from the driver's seat of the own vehicle 1. FIG. be. A virtual image V shown in FIG. and an AR virtual image V70. The AR virtual image V60 is displayed at a position (target position PT) corresponding to the position of the real object 300 present in the real scene. The AR virtual image V60 is displayed, for example, at a position superimposed on the real object 300 or in the vicinity of the real object 300, and notifies the existence of the real object 300 with emphasis. In other words, the “position corresponding to the position of the real object 300 (target position PT)” is not limited to the position superimposed on the real object 300 and viewed by the observer. There may be. Note that the AR virtual image V60 is arbitrary as long as it does not hinder the visual recognition of the real object 300 .

The AR virtual images V60 shown in FIG. 9 include navigation virtual images V61 and V62 that indicate guidance routes, enhanced virtual images V64 and V65 that highlight and notify attention targets, and POI virtual images V65 that indicate targets, predetermined buildings, and the like. . The position (target position PT) corresponding to the position of the real object 300 is the position of the road surface 311 (an example of the real object 300) on which these are superimposed in the navigation virtual images V61 and V62. (an example of the real object 300), in the enhanced virtual image V64 the position near the other vehicle 314 (an example of the real object 300), and in the POI virtual image V65 a building 315 (an example of the real object 300). ). As described above, the display control device 30 (processor 33) increases the correction amount C associated with the movement of the observer's eye position in the motion parallax adding process as the perceived distance D30 set for the virtual image V increases. That is, assuming that the perceptual distance D30 set for the virtual image V shown in FIG. The associated correction amount C is set as follows: correction amount of V65>correction amount of V64>correction amount of V63>correction amount of V62>correction amount of V61. Note that the virtual image V62 and the virtual image V61 are of the same kind and are displayed close to each other. The same correction amount may be set.

Further, the display control device 30 (processor 33) of some embodiments may set the correction amount C associated with the movement of the observer's eye position to zero in the non-AR virtual image V70 ( need not be corrected accordingly).

Further, the display control device 30 (processor 33) of some embodiments may correct the non-AR virtual image V70 according to the movement of the observer's eye position. In the example shown in FIG. 9, the non-AR virtual images V70 (V71, V72) are arranged below the virtual image display area VS, and the area of the road surface 311, which is the real object 300, overlaps with the navigation virtual image V61 of FIG. It is closer to the own vehicle 1 than the area of the road surface 311 . That is, the display control device 30 (processor 33) of some embodiments sets the perceived distance D30 of the non-AR virtual image V70 (V71, V72) to the AR virtual image V60 (strictly speaking, the lowest distance among the AR virtual images V60). ) is set shorter than the perception distance D30 of the navigation virtual image V61) arranged in ( In a narrow sense, it may be set to be smaller than the correction amount C of the navigation virtual image V61 positioned at the lowest position among the AR virtual images V60.

FIG. 10 is a block diagram of a vehicular virtual image display system, according to some embodiments. The display controller 30 comprises one or more I/O interfaces 31 , one or more processors 33 , one or more image processing circuits 35 and one or more memories 37 . Various functional blocks depicted in FIG. 3 may be implemented in hardware, software, or a combination of both. FIG. 10 is only one embodiment and the illustrated components may be combined into fewer components or there may be additional components. For example, image processing circuitry 35 (eg, a graphics processing unit) may be included in one or more processors 33 .

As shown, processor 33 and image processing circuitry 35 are operatively coupled with memory 37 . More specifically, the processor 33 and the image processing circuit 35 execute a program stored in the memory 37 to generate and/or transmit image data, for example, the vehicle display system 10 (display device). 40) can be controlled. Processor 33 and/or image processing circuitry 35 may include at least one general purpose microprocessor (e.g., central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA). , or any combination thereof. Memory 37 includes any type of magnetic media such as hard disks, any type of optical media such as CDs and DVDs, any type of semiconductor memory such as volatile memory, and non-volatile memory. Volatile memory may include DRAM and SRAM, and non-volatile memory may include ROM and NVRAM.

As shown, processor 33 is operatively coupled with I/O interface 31 . The I/O interface 31 communicates (CAN communication). The communication standard adopted by the I/O interface 31 is not limited to CAN. : MOST is a registered trademark), a wired communication interface such as UART or USB, or a personal area network (PAN) such as a Bluetooth network, a local network such as an 802.11x Wi-Fi network. It includes an in-vehicle communication (internal communication) interface, which is a short-range wireless communication interface within several tens of meters such as an area network (LAN). In addition, the I / O interface 31 is a wireless wide area network (WWAN0, IEEE802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE802.16e base (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, An external communication (external communication) interface such as a wide area communication network (for example, Internet communication network) may be included according to a cellular communication standard such as 5G.

As shown, the processor 33 is interoperably coupled with the I/O interface 31 to communicate with various other electronic devices and the like connected to the vehicle display system 10 (I/O interface 31). can be given and received. The I/O interface 31 includes, for example, a vehicle ECU 401, a road information database 403, a vehicle position detection unit 405, an operation detection unit 407, an eye position detection unit 409, an external sensor 411, a brightness detection unit 413, an IMU 415, portable information A terminal 417, an external communication device 419, etc. are operatively coupled. Note that the I/O interface 31 may include a function of processing (converting, calculating, and analyzing) information received from other electronic devices or the like connected to the vehicle display system 10 .

Display 40 is operatively coupled to processor 33 and image processing circuitry 35 . Accordingly, the image displayed by light modulating element 51 may be based on image data received from processor 33 and/or image processing circuitry 35 . The processor 33 and image processing circuit 35 control the image displayed by the light modulation element 51 based on the information obtained from the I/O interface 31 .

The vehicle ECU 401 detects the state of the vehicle 1 (for example, the ON/OFF state of the activation switch (eg, accessory switch: ACC or ignition switch: IGN) from sensors and switches provided in the vehicle 1 (an example of activation information. ), mileage, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine speed, motor speed, steering angle, shift position, drive mode, various warning conditions, attitude (including roll angle and/or pitching angle), vehicle vibration (including magnitude, frequency, and/or frequency of vibration)), etc., collecting and managing (controlling) the state of the own vehicle 1 ), and as part of its function, it outputs a signal indicating the numerical value of the state of the vehicle 1 (for example, the vehicle speed of the vehicle 1) to the processor 33 of the display control device 30. be able to. The vehicle ECU 401 simply transmits a numerical value detected by a sensor or the like (for example, the pitching angle is 3 [degrees] in the forward tilting direction) to the processor 33, or alternatively, transmits the numerical value detected by the sensor. determination result based on one or more states of the own vehicle 1 including, for example, that the own vehicle 1 satisfies a predetermined forward lean condition, or/and an analysis result (for example, brake pedal open In combination with the degree information, the fact that the vehicle is leaning forward due to braking) may be transmitted to the processor 33 . For example, the vehicle ECU 401 may output to the display control device 30 a signal indicating a determination result that the host vehicle 1 satisfies a predetermined condition stored in advance in a memory (not shown) of the vehicle ECU 401 . Note that the I/O interface 31 may acquire the above-described information from sensors and switches provided in the own vehicle 1 without using the vehicle ECU 401 .

In addition, the vehicle ECU 401 may output to the display control device 30 an instruction signal that instructs an image to be displayed by the vehicle display system 10. At this time, the coordinates, size, type, display mode of the image, the need for notification of the image, and so on. Necessity-related information that serves as a basis for determining the degree and/or the necessity of notification may be added to the instruction signal and transmitted.

The road information database 403 is included in a navigation device (not shown) provided in the own vehicle 1 or an external server connected to the own vehicle 1 via an external communication interface (I/O interface 31). Based on the position of the vehicle 1 acquired from the position detection unit 405, road information (lanes, white lines, stop line, pedestrian crossing, road width, number of lanes, intersection, curve, branch road, traffic regulation, etc.), presence/absence of feature information (buildings, bridges, rivers, etc.), position (including distance to own vehicle 1) ), direction, shape, type, detailed information, etc. may be read and sent to the processor 33 . The road information database 403 may also calculate an appropriate route (navigation information) from the departure point to the destination and output to the processor 33 a signal indicating the navigation information or image data indicating the route.

The vehicle position detection unit 405 is a GNSS (global navigation satellite system) or the like provided in the vehicle 1, detects the current position and direction of the vehicle 1, and sends a signal indicating the detection result to the processor 33. The information is output to the road information database 403 , a portable information terminal 417 and/or an external communication device 419 to be described later, via or directly. The road information database 403, a portable information terminal 417 (to be described later), and/or an external communication device 419 acquire position information of the vehicle 1 from the vehicle position detection unit 405 continuously, intermittently, or at each predetermined event. , information about the surroundings of the own vehicle 1 may be selected/generated and output to the processor 33 .

The operation detection unit 407 is, for example, a CID (Center Information Display) of the vehicle 1, a hardware switch provided on an instrument panel, or a software switch combining an image and a touch sensor. It outputs to the processor 33 the operation information based on the operation by one passenger (the user sitting in the driver's seat and/or the user sitting in the passenger's seat). For example, the operation detection unit 407 sets the display area setting information based on the operation of moving the virtual image display area 100, the eye box setting information based on the operation of moving the eye box 200, and the observer's eye position 700 by the user's operation. Information based on the operation to be performed is output to the processor 33 .

The eye position detection unit 409 includes a camera such as an infrared camera that detects the eye position 700 (see FIG. 1) of the observer sitting in the driver's seat of the vehicle 1, and outputs the captured image to the processor 33. good too. The processor 33 acquires a captured image (an example of information that can estimate the eye position 700) from the eye position detection unit 409, and analyzes the captured image by a method such as pattern matching to determine the eye positions of the observer. The coordinates of 700 may be detected and a signal indicative of the detected coordinates of eye position 700 may be output to processor 33 .

In addition, the eye position detection unit 409 obtains an analysis result obtained by analyzing the image captured by the camera (for example, where the observer's eye position 700 belongs in a spatial region corresponding to a plurality of preset display parameters). ) may be output to the processor 33 . Note that the method of acquiring the eye position 700 of the observer of the own vehicle 1 or the information that can estimate the eye position 700 of the observer is not limited to these, and a known eye position detection (estimation) technique can be used. may be obtained using

Further, the eye position detection unit 409 detects the moving speed and/or moving direction of the observer's eye position 700, and outputs a signal indicating the moving speed and/or moving direction of the observer's eye position 700 to the processor 33. can be output to

Also, the eye position detection unit 409 may have a function as the line-of-sight direction detection unit 409 . The line-of-sight direction detection unit 409 may include an infrared camera or a visible light camera that captures the face of an observer sitting in the driver's seat of the vehicle 1 , and may output the captured image to the processor 33 . The processor 33 acquires a captured image (an example of information for estimating the line-of-sight direction) from the line-of-sight direction detection unit 409, and analyzes the captured image to determine the line-of-sight direction (and/or the gaze position) of the observer. can be specified. Note that the line-of-sight direction detection unit 409 may analyze the captured image from the camera and output to the processor 33 a signal indicating the line-of-sight direction (and/or the gaze position) of the observer, which is the analysis result. It should be noted that the method of acquiring information that can estimate the line-of-sight direction of the observer of the own vehicle 1 is not limited to these methods, and includes an EOG (Electro-oculogram) method, a corneal reflection method, a scleral reflection method, a Purkinje image It may also be obtained using other known gaze direction detection (estimation) techniques such as detection methods, search coil methods, infrared fundus camera methods, and the like.

The vehicle exterior sensor 411 detects real objects existing around the vehicle 1 (front, side, and rear). Real objects detected by the sensor 411 outside the vehicle include, for example, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), road surfaces of driving lanes, lane markings, roadside objects, and/or features (buildings, etc.), which will be described later. and so on. Exterior sensors include, for example, radar sensors such as millimeter wave radar, ultrasonic radar, and laser radar, cameras, or a detection unit consisting of a combination thereof, and processing detection data from the one or more detection units ( data fusion) and a processing device. A conventional well-known method is applied to object detection by these radar sensors and camera sensors. By object detection by these sensors, the presence or absence of a real object in the three-dimensional space, and if the real object exists, the position of the real object (relative distance from the own vehicle 1, the traveling direction of the own vehicle 1) Horizontal position, vertical position, etc.), size (horizontal direction (horizontal direction), height direction (vertical direction), etc.), movement direction (horizontal direction (horizontal direction) , depth direction (front-rear direction)), movement speed (horizontal direction (left-right direction), depth direction (front-rear direction)), and/or type and the like may be detected.

One or a plurality of vehicle exterior sensors 411 detects a real object in front of the own vehicle 1 at each detection cycle of each sensor, and obtains real object information (presence or absence of a real object, whether or not a real object exists), which is an example of real object information. Information such as the position, size and/or type of each real object, if any, can be output to the processor 33 . Note that the real object information may be transmitted to the processor 33 via another device (for example, the vehicle ECU 401). Also, when using a camera as a sensor, an infrared camera or a near-infrared camera is desirable so that a real object can be detected even when the surroundings are dark, such as at night. Moreover, when using a camera as a sensor, a stereo camera that can acquire distance and the like by parallax is desirable. Here, the relative distance of the detected real object 300 from the own vehicle 1 is defined as an object distance (distance to the real object) D4.

The brightness detection unit 413 converts the illuminance or luminance of a predetermined range of the foreground that exists in front of the passenger compartment of the vehicle 1 into the outside world brightness (an example of brightness information), or converts the illuminance or luminance in the passenger compartment into the vehicle interior brightness ( (an example of brightness information). The brightness detection unit 413 is, for example, a phototransistor or a photodiode, and is mounted on the instrument panel, the room mirror, the HUD device 20, or the like of the own vehicle 1 shown in FIG.

The IMU 415 includes one or more sensors (e.g., accelerometers and gyroscopes) configured to detect the position, orientation, and changes thereof (velocity of change, acceleration of change) of the host vehicle 1 based on inertial acceleration. ). The IMU 415 analyzes the detected values (the detected values include signals indicating the position and orientation of the host vehicle 1 and their changes (change speed and change acceleration)), the results of analyzing the detected values, It may be output to processor 33 . The analysis result is a signal or the like indicating whether or not the detected value satisfies a predetermined condition. Therefore, the signal may be a signal indicating that the behavior (vibration) of the own vehicle 1 is small.

The mobile information terminal 417 is a smart phone, a notebook computer, a smart watch, or other information equipment that can be carried by the observer (or other occupant of the own vehicle 1). By pairing with the mobile information terminal 417, the I/O interface 31 can communicate with the mobile information terminal 417, and can read data recorded in the mobile information terminal 417 (or a server through the mobile information terminal). to get The portable information terminal 417 has, for example, the same functions as the road information database 403 and the vehicle position detection unit 405 described above, acquires the road information (an example of real object related information), and transmits it to the processor 33. may Also, the mobile information terminal 417 may acquire commercial information (an example of real object related information) related to commercial facilities near the own vehicle 1 and transmit it to the processor 33 . In addition, the mobile information terminal 417 transmits schedule information of the owner of the mobile information terminal 417 (for example, an observer), incoming call information at the mobile information terminal 417, mail reception information, etc. to the processor 33, and the processor 33 and the image Processing circuitry 35 may generate and/or transmit image data for these.

The external communication device 419 is a communication device that exchanges information with the own vehicle 1, for example, other vehicles connected to the own vehicle 1 by vehicle-to-vehicle communication (V2V: vehicle-to-vehicle), vehicle-to-pedestrian communication (V2P: vehicle To Pedestrian) connected by pedestrians (mobile information terminals carried by pedestrians), road-to-vehicle communication (V2I: Vehicle To roadside Infrastructure) is a network communication device connected by Includes everything connected by communication (V2X: Vehicle To Everything). The external communication device 419 acquires, for example, the positions of pedestrians, bicycles, motorcycles, other vehicles (preceding vehicles, etc.), road surfaces, lane markings, roadside objects, and/or features (buildings, etc.), and sends them to the processor 33. may be output. Further, the external communication device 419 has the same function as the vehicle position detection unit 405 described above, and may acquire the position information of the vehicle 1 and transmit it to the processor 33. Further, the road information database 403 and may acquire the road information (an example of real object related information) and transmit it to the processor 33 . Information acquired from the external communication device 419 is not limited to the above.

The software components stored in the memory 37 include an eye position detection module 502, an eye position estimation module 504, an eye position prediction module 506, an eye position state determination module 508, a vehicle state determination module 510, an eye-following image processing module 512, It includes a graphics module 514, a light source drive module 516, an actuator drive module 518, and the like.

11A, 11B, and 11C are flow diagrams illustrating a method S100 for performing image correction operations based on observer eye position. The method S100 is performed in a HUD device 20 that includes a display and a display controller 30 that controls the HUD device 20 . Some of the acts of method S100 described below are optionally combined, some of the steps of the acts are optionally varied, and some of the acts are optionally omitted.

First, the display control device 30 (processor 33) detects the eye position 700 of the observer (step S110).

(Step S112)
In step S110 in some embodiments, the display control device 30 (processor 33) executes the eye position detection module 502 of FIG. Detect (acquire information indicating the eye position 700). The eye position detection module 502 detects the coordinates indicating the observer's eye position 700 (positions in the X and Y axis directions and is an example of a signal indicating the eye position 700), and detects the height of the observer's eye. coordinates indicating the height (position in the Y-axis direction, which is an example of a signal indicating the eye position 700); and/or coordinates indicating the observer's eye position 700 (which are positions in the X, Y, and Z axis directions and are an example of a signal indicating the eye position 700). 700 includes various software components for performing various operations related to detecting.

Note that the eye position 700 detected by the eye position detection module 502 includes positions 700R and 700L of the right and left eyes, one of the predetermined positions of the right eye position 700R and the left eye position 700L, the right eye position 700R and the left eye position 700L. Detectable (easily detectable) position, or a position calculated from the right eye position 700R and the left eye position 700L (for example, the middle point between the right eye position and the left eye position), etc. good. For example, the eye position detection module 502 determines the eye position 700 based on the observation position acquired from the eye position detection unit 409 immediately before the timing of updating the display settings.

Further, the eye position detection unit 409 detects the movement direction and/or the movement speed of the observer's eye position 700 based on a plurality of observation positions with different detection timings of the observer's eyes obtained from the eye position detection unit 409. A signal indicating the moving direction and/or moving speed of the observer's eye position 700 may be output to the processor 33 .

(Step S114)
Further, in step S110 in some embodiments, the display control device 30 (processor 33) may acquire information that enables eye position estimation by executing the eye position estimation module 504 (step S114). . Information from which the eye positions can be estimated is, for example, the captured image acquired from the eye position detection unit 409, the position of the driver's seat of the own vehicle 1, the position of the observer's face, the height of the sitting height, or the eyes of a plurality of observers. such as the observation position of The eye position estimating module 504 estimates the eye position 700 of the observer of the own vehicle 1 from the information that can estimate the eye position. The eye position estimation module 504 uses the captured image acquired from the eye position detection unit 409, the position of the driver's seat of the own vehicle 1, the position of the face of the observer, the height of the sitting height, or the observation positions of the eyes of a plurality of observers. includes various software components for performing various operations related to estimating the observer's eye position 700, such as estimating the observer's eye position 700 from the. That is, the eye position estimation module 504 can include table data, arithmetic expressions, and the like for estimating the eye position 700 of the observer from information that can estimate the eye position.

(Step S116)
Further, in step S110 in some embodiments, the display control device 30 (processor 33) may acquire information that can predict the eye position 700 of the observer by executing the eye position prediction module 506. (Step S116). The information that can predict the eye position 700 of the observer is, for example, the latest observation position obtained from the eye position detection unit 409, one or more observation positions obtained in the past, or the like. Eye position prediction module 506 includes various software components for performing various operations related to predicting eye positions 700 based on information capable of predicting eye positions 700 of an observer. Specifically, for example, the eye position prediction module 506 predicts the eye position 700 of the observer at the timing when the observer visually recognizes the image to which the new display settings are applied. The eye position prediction module 506 uses one or more past observed positions, for example, using a least squares method, a prediction algorithm such as a Kalman filter, an α-β filter, or a particle filter to calculate the next value. You can make a prediction.

(Step S120)
Next, the display control device 30 (processor 33) determines whether a predetermined condition is satisfied (step S120).

FIG. 11B is a diagram explaining the operation of the eye position state determination module 508 in step S130.

(Step S130)
In step S120 in some embodiments, the display control device 30 (processor 33) executes the eye position state determination module 508 of FIG. It is determined whether the eye position 700 (movement of the eye position 700) satisfies a predetermined condition based on possible information or information from which the eye position can be predicted.

FIG. 12 shows (11) vertical eye position (or head position) Py (Y1, Y2, Y3. ..Y10), (12) Amount of change ΔPy (Py1 (=Y2-Y1), Py2 (=Y3-Y2), Py3 (=Y4-Y3), in vertical eye position (or head position), . . . Py9 (=Y10−Y9)), (13) Moving speed Vy (Vy1 (=Py1/Δt), Vy2 (=Py2/Δt), Vy3 of eye position (or head position) in the vertical direction (=Py3/Δt), . ) Amount of change in eye position (or head position) in the vertical direction ΔPx (Px1 (=X2-X1), Px2 (=X3-X2), Px3 (=X4-X3), ... Px9 (=X10 -X9)), and (23) Movement speed Vx (Vx1 (=Px1/Δt), Vx2 (=Px2/Δt), Vx3 (=Px3/Δt) of eye position (or head position may be used) in the vertical direction , . . . Vx9 (=Px9/Δt)).

(Step S131)
In step S130 in some embodiments, the display control device 30 (processor 33) executes the eye position state determination module 508 of FIG. , it may be determined that a predetermined condition is satisfied. For example, the eye position state determination module 508 determines that the amount of change ΔPx in the eye position in the horizontal direction shown in FIG. , it may be determined that the predetermined condition is satisfied.

(Step S132)
Further, in step S130 in some embodiments, the display control device 30 (processor 33) executes the eye position state determination module 508 of FIG. If so, it may be determined that a predetermined condition is satisfied. For example, the eye position state determination module 508 determines that the eye position change amount ΔPy in the vertical direction shown in FIG. , it may be determined that the predetermined condition is satisfied.

(Step S133)
Further, in step S130 in some embodiments, the display control device 30 (processor 33) can execute the distance determination module 511 of FIG. 10 to compare, for example, the object distance D4 and the imaging distance D10. Yes, and if the object distance D4 is greater than the imaging distance D10, it may be determined that the predetermined condition is satisfied.

(Step S134)
Further, in step S130 in some embodiments, the display control device 30 (processor 33) executes the distance determination module 511 of FIG. A second distance threshold (for example, 20 m) can be compared, and it may be determined that a predetermined condition is satisfied. Here, the second distance threshold is set to approximately five times the imaging distance D10.

FIG. 11C is a diagram illustrating the operation of vehicle state determination module 510 in step S140.

(Step S140)
In step S120 in some embodiments, the display control device 30 (processor 33) may determine whether the vehicle state satisfies a predetermined condition by executing the vehicle state determination module 510 of FIG. .

(Step S141)
In step S140 in some embodiments, the display control device 30 (processor 33) executes the vehicle state determination module 510 of FIG. information, information indicating the vehicle position obtained from the vehicle position detection unit 405, information obtained from the IMU 415, etc., it is estimated whether or not the vehicle 1 is running. When estimated, it may be determined that the predetermined condition is satisfied.

(Step S150)
Refer again to FIG. 11A. After it is determined in step S120 whether the predetermined condition is satisfied, the display control device 30 (processor 33) executes the eye-following image processing module 512 to determine the observer's eye position 700 (the eye position 700 is Specifically, the eye position Py in the vertical direction and the eye position Px in the horizontal direction in FIG. (eye-tracking image processing).

10 performs first image correction processing S160 (step S160), second image correction processing S170 (step S170), and third image correction processing S170 (step S170) based on the determination result in step S120. Switching between image correction processing S180 (step S180).

(First image correction process S160)
When it is determined in step S120 that the predetermined condition is satisfied (step S131), the eye-following image processing module 512 of FIG. The horizontal position of the virtual image V is corrected by a horizontal correction amount (first correction amount) Cx. The first left/right correction amount Cx (the same applies to a second left/right correction amount Cxq1 and a third left/right correction amount Cxq2, which will be described later) is a parameter that gradually increases as the amount of change ΔPx in the eye position in the left/right direction increases. is. In addition, the first left/right correction amount Cx (the same applies to a second left/right correction amount Cxq1 and a third left/right correction amount Cxq2, which will be described later) gradually increases as the perceived distance D30 set to the virtual image V increases. is a parameter. The first image correction processing S160 is to create an image that perfectly reproduces natural motion parallax as if the virtual image V were fixed at the set target position PT even when viewed from each of the left and right eye positions Px. This includes position correction, and in a broader sense, it may also include correction of the image position so as to approximate natural motion parallax. That is, the first image correction processing S160 sets the display position of the virtual image V to the position of the intersection of the virtual image display area VS and the straight line connecting the target position PT set in the virtual image V and the observer's eye position 700. Align (bring the display position of the virtual image V closer).

(First image correction process S160)
When it is determined in step S120 that the predetermined condition is satisfied (step S132), the eye-following image processing module 512 of FIG. The vertical position of the virtual image V is corrected by a vertical correction amount (first correction amount) Cy. The first vertical correction amount Cy (the same applies to a second vertical correction amount Cyr1 and a third vertical correction amount Cyr2, which will be described later) is a parameter that gradually increases as the eye position change amount ΔPy in the vertical direction increases. is. Also, the first vertical correction amount Cy (the same applies to a second vertical correction amount Cyr1 and a third vertical correction amount Cyr2, which will be described later) gradually increases as the perceived distance D30 set to the virtual image V increases. is a parameter. The first image correction processing S160 is to create an image that perfectly reproduces natural motion parallax as if the virtual image V were fixed at the set target position PT even when viewed from each eye position Py in the vertical direction. This includes position correction, and in a broader sense, it may also include correction of the image position so as to approximate natural motion parallax. That is, the first image correction processing S160 sets the display position of the virtual image V to the position of the intersection of the virtual image display area VS and the straight line connecting the target position PT set in the virtual image V and the observer's eye position 700. Align (bring the display position of the virtual image V closer).

(Second image correction processing S170)
When it is determined in step S120 that the predetermined condition is satisfied (step S133), the eye-following image processing module 512 of FIG. The horizontal position of the virtual image V is corrected by a horizontal correction amount (second correction amount) Cxq1. As shown in FIG. 13A, the second left/right correction amount Cxq1 is 0, which is 1 or less than the first left/right correction amount Cx with respect to the eye position change amount ΔPx in the left/right direction in the first image correction processing S160. It is a value obtained by multiplying 0.9 by a first left and right coefficient (first predetermined coefficient) Q1. Specifically, for example, if the first lateral correction amount Cx with respect to the eye position change amount ΔPx in the horizontal direction is 100%, the second lateral correction amount Cxq1 with respect to the eye position change amount ΔPx in the vertical direction is 90%. is. In a broad sense, the second left/right correction amount Cxq1 should be smaller than the first left/right correction amount Cx to suppress the first left/right correction amount Cx. %, but preferably less than 90% of the first lateral correction amount Cx.

(Second image correction processing S170)
When it is determined in step S120 that the predetermined condition is satisfied (step S133), the eye-following image processing module 512 of FIG. The vertical position of the virtual image V is corrected by a vertical correction amount (second correction amount) Cyr1. As shown in FIG. 13B, the second vertical correction amount Cyr1 is 0, which is 1 or less than the first vertical correction amount Cy for the eye position change amount ΔPy in the vertical direction in the first image correction processing S160. .8 multiplied by a first upper and lower coefficient (first predetermined coefficient) R1. Specifically, for example, when the first vertical correction amount Cy for the eye position change amount ΔPy in the vertical direction is 100%, the second vertical correction amount Cyr1 for the same vertical eye position change amount ΔPy is 80%. is. In a broad sense, the second vertical correction amount Cyr1 should be smaller than the first vertical correction amount Cy to suppress the first horizontal correction amount Cx. %, but preferably less than 80% of the first vertical correction amount Cy.

(Example of step S170)
The first left-right coefficient Q1 is preferably greater than the first up-down coefficient R1.

(Example of step S170)
In step S170 in some embodiments, the display control device 30 (processor 33) shifts to the second image correction process, as shown in FIG. The second left/right correction amount Cxq1 for ΔPx may be gradually decreased over time as the distance from the first left/right correction amount Cx for the eye position change amount ΔPx in the left/right direction increases (see Cxq11). For example, assuming that the first lateral correction amount Cx and the second lateral correction amount Cxq1 with respect to the amount of change ΔPx in the eye position in the lateral direction are 100% and 90%, respectively, the display control device 30 (processor 33) The left/right correction amount Cxq1 may be gradually decreased from the first left/right correction amount Cx over time, such as 100%→98%→96% . . . →92%. Likewise, the second vertical correction amount Cyr1 for the eye position change amount ΔPy in the vertical direction gradually increases over time as it becomes more distant from the first vertical correction amount Cy for the eye position change amount ΔPy in the vertical direction. can be reduced to

This time-varying correction amount Cxq11 gradually decreases over time as the object distance increases, and conversely increases over time as the object distance decreases. It may be different when the object distance becomes longer and when the object distance becomes closer.

(Third image correction processing S180)
When it is determined in step S120 that the predetermined condition is satisfied (step S134), the eye-following image processing module 512 of FIG. The horizontal position of the virtual image V is corrected by a horizontal correction amount (third correction amount) Cxq2. As shown in FIG. 13A, the third left/right correction amount Cxq2 is the first left/right correction amount Cx with respect to the eye position change amount ΔPx in the left/right direction in the first image correction processing S160, and the first left/right coefficient It is a value multiplied by a second left and right coefficient (second predetermined coefficient) Q2 of 0.5 which is smaller than Q1. Specifically, for example, if the first lateral correction amount Cx for the eye position change amount ΔPx in the lateral direction is 100%, the third lateral correction amount Cxq2 for the same eye position change amount ΔPx in the vertical direction is 50%. is. In a broad sense, the third left/right correction amount Cxq2 should be smaller than the first left/right correction amount Cx to suppress the first left/right correction amount Cx. %, but preferably less than 50% of the first lateral correction amount Cx.

(Third image correction processing S180)
When it is determined in step S120 that the predetermined condition is satisfied (step S134), the eye-following image processing module 512 of FIG. The vertical position of the virtual image V is corrected by a vertical correction amount (third correction amount) Cyr2. As shown in FIG. 13B, the third vertical correction amount Cyr2 is 0.4 times the first vertical correction amount Cy for the eye position change amount ΔPy in the vertical direction in the first image correction processing S160. It is a value obtained by multiplying 1 by a second upper/lower coefficient (second predetermined coefficient) R2 smaller than the upper/lower coefficient R1. Specifically, for example, when the first vertical correction amount Cy for the eye position change amount ΔPy in the vertical direction is 100%, the third vertical correction amount Cyr2 for the same eye position change amount ΔPy in the vertical direction is 40%. is. In a broad sense, the third vertical correction amount Cyr2 should be smaller than the first vertical correction amount Cy to suppress the first horizontal correction amount Cx. %, but preferably less than 40% of the first vertical correction amount Cy.

(Example of step S180)
In step S180 in some embodiments, the display control device 30 (processor 33), as shown in FIG. The third left/right correction amount Cxq2 for ΔPx may gradually decrease over time so as to separate from the second left/right correction amount Cxq1 for the eye position change amount ΔPx in the left/right direction (see Cxq21). For example, assuming that the second lateral correction amount Cxq1 is 90% and the third lateral correction amount Cxq2 is 50% with respect to the eye position change amount ΔPx in the lateral direction, the display control device 30 (processor 33) The left/right correction amount Cxq2 may be gradually decreased over time from the second left/right correction amount Cxq1 in the manner of 90%→80%→70%→ . . . →50%. Likewise, the third vertical correction amount Cyr2 for the eye position change amount ΔPy in the vertical direction is also gradually separated from the second vertical correction amount Cyr1 for the eye position change amount ΔPy in the vertical direction. You can make it smaller.

This time-varying correction amount Cxq21 gradually decreases over time as the object distance increases, and conversely increases over time as the object distance decreases. It may be different when the object distance becomes longer and when the object distance becomes closer.

(Step S190)
In step S160 in some embodiments, when the display control device 30 (processor 33) determines that the predetermined cancellation condition is satisfied, the process shifts from the second image correction processing S170 to the first image correction processing S160. .

The predetermined cancellation condition includes that a predetermined period of time (for example, 20 seconds) has passed since the second image correction process S170 was started. The eye-tracking image processing module 512 executes timekeeping after shifting to the second image correction processing S170, and the predetermined time stored in advance in the memory 37 (or set by the operation detection unit 407) has elapsed. If so, it may be determined that the cancellation condition is satisfied.

(Step S190)
In step S160 in some embodiments, when the display control device 30 (processor 33) determines that the predetermined cancellation condition is satisfied, the process moves from the third image correction process S180 to the second image correction process S170. .

The predetermined cancellation condition includes that a predetermined period of time (for example, 20 seconds) has passed since the third image correction process S180 was started. The eye-following image processing module 512 executes timekeeping after shifting to the third image correction processing S180, and the predetermined time stored in advance in the memory 37 (or set by the operation detection unit 407) has elapsed. If so, it may be determined that the cancellation condition is satisfied.

Further, the predetermined cancellation condition may include that the predetermined condition is no longer satisfied in step S120. That is, the predetermined cancellation condition is detecting that at least one of steps S131 to S134 and step S141 has changed from a state in which the predetermined condition is satisfied to a state in which the predetermined condition is no longer satisfied; may contain Further, the predetermined cancellation condition may include that a predetermined time (for example, 20 seconds) has passed after the predetermined condition is no longer satisfied in step S120.

Again, refer to FIG. Graphics module 514 of FIG. 10 includes various known software components for performing image processing, such as rendering, to generate image data, and to drive display 40 . The graphic module 514 also controls the type (moving image, still image, shape), arrangement (positional coordinates, angle), size, display distance (in the case of 3D), visual effects (for example, brightness, (transparency, saturation, contrast, or other visual properties) may be included. The graphic module 514 stores the type of image (one example of display parameters), the position coordinates of the image (one example of display parameters), the angle of the image (pitch angle with the X direction as the axis, Y direction as the axis, yaw rate angle around the axis, rolling angle around the Z direction, etc., which are examples of display parameters), image size (one example of display parameters), image color (hue, saturation , one example of display parameters set by brightness, etc.), and the intensity of perspective expression of an image (one of display parameters set by the position of a vanishing point, etc.) to make an image visible to the observer. Data may be generated to drive the light modulating element 50 .

Light source driving module 516 includes various known software components for performing driving of light source unit 24 . The light source driving module 516 may drive the light source units 24 based on the set display parameters.

Actuator drive module 518 includes various known software components for performing driving first actuator 28 and/or second actuator 29. One actuator 28 and a second actuator 29 can be driven.

As described above, the display control device 30 of this embodiment includes a display (40) for displaying an image, and a relay optical system for projecting the light of the image displayed by the display (40) onto a projection target member. (80) for executing display control in a head-up display device (20) that allows a user of the vehicle to visually recognize a virtual image of the image superimposed on the foreground, comprising one or more a processor (33) of, a memory (37) and one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33); , a vehicle exterior sensor (411) that detects a real object existing around the vehicle, and the processor (33) detects the user's eye position and/or head position (Py) in the vertical direction of the vehicle, and Acquire the eye position and/or head position (Px) in the horizontal direction of the vehicle, and change (ΔPy) in the eye position or head position (Py) in the vertical direction and the eye position or head position (Px) in the horizontal direction a first image correction process (S160) for correcting the position of the image displayed on the display (40) with a first correction amount (Cx, Cy) based at least on the display (40); A second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or less than 1. and executing the first image correction process (S170) when the distance (D4) from the vehicle to the real object in the foreground is equal to or less than a first distance threshold. Then, when the distance (D4) from the vehicle to the real object in the foreground is greater than the first distance threshold, the second image correction process (S170) is executed.
As a result, even if it is detected (erroneous detection) that the eye position of the observer has moved by a movement amount greater than or equal to the predetermined movement amount, the correction of the position of the image is suppressed, thereby preventing the observer from feeling discomfort. can be reduced.

Further, in some embodiments, the first distance threshold may be a distance (D10) from the vehicle to the area where the virtual image is displayed.

Further, in some embodiments, the position of the image displayed on the display (40) is the third correction amount obtained by multiplying the first correction amount (Cx, Cy) by a second predetermined coefficient (Q2, R2). and a third image correction process (S180) for correcting with the correction amount (Cxq2, Cyr2) of, and when the distance (D4) to the real object is larger than the second distance threshold, the 3 image correction processing (S180) may be executed.

Also, in some embodiments, the second distance threshold may be greater than the first distance threshold.

Also, in some embodiments, the second predetermined coefficient (Q2, R2) may be smaller than the first predetermined coefficient (Q1, R1).

Further, in some embodiments, when the position of the image displayed on the display (40) is corrected based on the change (ΔPy) in the eye position or head position (Px) in the horizontal direction, the first The predetermined coefficient (Q1) of is the first coefficient (Q1) when correcting the position of the image displayed on the display (40) based on the change (ΔPy) in the eye position or head position (Py) in the vertical direction. It may be larger than the predetermined coefficient (R1).

Further, in some embodiments, the first predetermined coefficient (Q1) in the second image correction process is the second correction amount (Cxq1, Cyr1) in the first image correction process. The first correction amount (Cx, Cy) may change over time as the distance from the first correction amount (Cx, Cy) increases.

In some embodiments, the head-up display device 20 includes a display (40) that displays an image, and a relay optical system ( 80), and performs display control in a head-up display device (20) that allows a vehicle user to visually recognize a virtual image of the image superimposed on the foreground, wherein one or more a processor (33), a memory (37), one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33); a vehicle exterior sensor (411) that detects a real object existing around the vehicle; The eye position and/or head position (Px) in the horizontal direction of the vehicle is obtained, and the eye position or head position (Py) in the vertical direction and the change (ΔPy) in the eye position or head position (Px) in the horizontal direction a first image correction process (S160) for correcting the position of the image displayed on the display (40) with a first correction amount (Cx, Cy) based on at least; a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) of 1 or less; and image correction processing (S170), wherein the first image correction processing is performed when a distance (D4) from the vehicle to the real object in the foreground is equal to or less than a first distance threshold. and executing the second image correction process (S170) when the distance (D4) from the vehicle to the real object in the foreground is greater than the first distance threshold.

The operations of the processing processes described above may be implemented by executing one or more functional modules of an information processing device such as a general purpose processor or an application specific chip. These modules, combinations of these modules, and/or combinations with known hardware that can replace their functions are all within the scope of protection of the present invention.

The functional blocks of vehicle display system 10 are optionally implemented in hardware, software, or a combination of hardware and software to carry out the principles of the various described embodiments. It is noted that the functional blocks illustrated in FIG. 10 may optionally be combined or separated into two or more sub-blocks to implement the principles of the described embodiments. will be understood by those skilled in the art. Accordingly, the description herein optionally supports any possible combination or division of the functional blocks described herein.

Reference Signs List 1: Vehicle 2: Projected portion 5: Dashboard 6: Road surface 10: Vehicle display system 20: HUD device 21: Light exit window 22: Housing 24: Light source unit 28: First actuator 29: Second actuator 30: Display control device 31 : I/O interface 33 : Processor 35 : Image processing circuit 37 : Memory 40 : Display device 205 : Center 300 : Real object 311 : Road surface 313 : Person 314 : Other vehicle 315 : Building 401 : Vehicle ECU
403 : road information database 405 : own vehicle position detection unit 407 : operation detection unit 409 : eye position detection unit 411 : outside sensor 413 : brightness detection unit 417 : portable information terminal 419 : external communication device 502 : eye position detection module 504 : Eye position estimation module 506 : Eye position prediction module 508 : Eye position state determination module 510 : Vehicle state determination module 511 : Distance determination module 512 : Eye-following image processing module 514 : Graphic module 516 : Light source drive module 518 : Actuator drive Module 710: Head position Cx: First lateral correction amount (first correction amount)
Cxq1: Second left/right correction amount (second correction amount)
Cxq2: Third right/left correction amount (third correction amount)
Cy: first vertical correction amount (first correction amount)
Cyr1: Second vertical correction amount (second correction amount)
Cyr2: third vertical correction amount (third correction amount)
Q1: first left-right coefficient (first predetermined coefficient)
Q2: second left-right coefficient (second predetermined coefficient)
R1: first upper and lower coefficient (first predetermined coefficient)
R2: second upper and lower coefficient (second predetermined coefficient)
D10: imaging distance (first distance threshold)
D30: Perceived distance D4: Object distance (distance to real object)
FU: virtual object (perceptual virtual image)
K: display light PS: virtual image display area PT: target position Px: eye position (head position) in the horizontal direction
Py: Vertical eye position (head position)
V: virtual image V60: AR virtual image (upper virtual image)
V61: navigation virtual image V62: navigation virtual image V63: enhanced virtual image V64: enhanced virtual image V65: enhanced virtual image V65: POI virtual image V70: non-AR virtual image (lower virtual image)
VS: virtual image display area Vx: moving speed Vy: moving speed t: cycle time ΔPx: amount of change ΔPy: amount of change

Claims (8)

It comprises at least a display (40) for displaying an image, and a relay optical system (80) for projecting the light of the image displayed by the display (40) onto a projection target member, and presents a virtual image of the image to the user of the vehicle. A display control device (30) that executes display control in a head-up display device (20) that is superimposed on the foreground and visually recognized,
one or more processors (33);
a memory (37);
one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33);
a vehicle exterior sensor (411) that detects a real object existing around the vehicle;
The processor (33) acquires the user's eye position and/or head position (Py) in the vertical direction of the vehicle and the eye position and/or head position (Px) in the horizontal direction of the vehicle. displayed on the display (40) with a first correction amount (Cx, Cy) based at least on the eye position or head position (Py) and the change (ΔPy) in the eye position or head position (Px) in the horizontal direction a first image correction process (S160) for correcting the position of the image to be
A second correction amount (Cxq1 , Cyr1) and a second image correction process (S170),
performing the first image correction process when a distance (D4) from the vehicle to the real object in the foreground is equal to or less than a first distance threshold;
executing the second image correction process (S170) when the distance (D4) from the vehicle to the real object in the foreground is greater than the first distance threshold;
A display controller (30). The first distance threshold is the distance (D10) from the vehicle to the area where the virtual image is displayed,
A display control device (30) according to claim 1, characterized in that: The position of the image displayed on the display (40) is determined by a third correction amount (Cxq2, Cyr2) obtained by multiplying the first correction amount (Cx, Cy) by a second predetermined coefficient (Q2, R2). and a third image correction process (S180) to correct,
executing the third image correction process (S180) when the distance (D4) to the real object is greater than a second distance threshold;
A display control device (30) according to claim 1, characterized in that: the second distance threshold is greater than the first distance threshold;
A display control device (30) according to claim 3, characterized in that: the second predetermined coefficient (Q2, R2) is smaller than the first predetermined coefficient (Q1, R1);
A display control device (30) according to claim 3, characterized in that: The first predetermined coefficient (Q1) when correcting the position of the image displayed on the display (40) based on the change (ΔPy) in the eye position or the head position (Px) in the horizontal direction is larger than the first predetermined coefficient (R1) when correcting the position of the image displayed on the display (40) based on a change (ΔPy) in eye position or head position (Py) in the vertical direction ,
A display control device (30) according to claim 1, characterized in that: In the second image correction process, the first predetermined coefficient (Q1) is
The second correction amount (Cxq1, Cyr1) changes over time as it moves away from the first correction amount (Cx, Cy) in the first image correction process.
A display control device (30) according to claim 1, characterized in that: It comprises at least a display (40) for displaying an image, and a relay optical system (80) for projecting the light of the image displayed by the display (40) onto a projection target member, and presents a virtual image of the image to the user of the vehicle. A display control device (30) that executes display control in a head-up display device (20) that is superimposed on the foreground and visually recognized,
one or more processors (33);
a memory (37);
one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33);
a vehicle exterior sensor (411) that detects a real object existing around the vehicle;
The processor (33) acquires the user's eye position and/or head position (Py) in the vertical direction of the vehicle and the eye position and/or head position (Px) in the horizontal direction of the vehicle. displayed on the display (40) with a first correction amount (Cx, Cy) based at least on the eye position or head position (Py) and the change (ΔPy) in the eye position or head position (Px) in the horizontal direction a first image correction process (S160) for correcting the position of the image to be
A second correction amount (Cxq1 , Cyr1) and a second image correction process (S170),
performing the first image correction process when a distance (D4) from the vehicle to the real object in the foreground is equal to or less than a first distance threshold;
executing the second image correction process (S170) when the distance (D4) from the vehicle to the real object in the foreground is greater than the first distance threshold;
A head-up display device (20).

Images