JP2022113292A - Display control device, head-up display device, and display control method - Google Patents

Abstract

To display an image which makes an observer easily sense depth feeling.SOLUTION: In a head-up display device for enabling an observer to visually recognize a virtual image of an image while overlapping a front scenery, a processor performs thickness display processing of displaying the virtual image as one or a plurality of perspective virtual images with depth expression having a thickness VT in a vertical direction when viewed from the observer and gradually reducing the thickness VT so as to converge to a disappearance point VP20, and the thickness disappearance point VP20 is arranged below a disappearance point VP0 of the front scenery when viewed from the observer.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to a display control device, a head-up display device, and a head-up display device that are used in a mobile object such as a vehicle and superimpose an image on the foreground of the mobile object (actual view in the forward direction of the mobile object seen from the vehicle occupant). It relates to a display control method and the like.

In Patent Document 1, display light projected onto a projection target portion such as a front windshield of a vehicle is reflected toward an occupant (observer) of the vehicle inside the vehicle, so that the observer can see the vehicle. describes a head-up display device (an example of a virtual image display device) that allows a user to visually recognize a virtual image that overlaps the foreground of the image. In particular, the head-up display device described in Patent Literature 1 virtualizes a display object (a A virtual image) is placed, and even if there is a change in the attitude of the vehicle or the position of the observer's eyes, the display object will appear as if it were at the target position in the foreground. It controls the image displayed inside the display device. That is, such a head-up display device creates an augmented reality in which a virtual object is added to a real landscape (foreground) and displayed.

In the example of FIG. 4 of Patent Document 1, an arrow pointing forward (in other words, extending from the rear to the front) is placed at a position slightly floating from the road surface on which the vehicle is traveling (a position away from the road surface in the upward direction). A virtual object in the shape of an arrow is displayed. In such a case, the observer will notice that the width at the rear end of the arrow (typically, the width in the left-right direction) gradually narrows toward the tip, as in the example of FIG. 5 of Patent Document 1. , the sense of perspective is expressed. That is, it is perceived that the trailing end of the arrow is near and the tip of the arrow is distant. A method such as a linear perspective method (perspective drawing method) is typically used to express such perspective.

JP 2010-156608 A

By the way, in the head-up display device as described above, the virtual image displayable area is typically a flat or rectangular area (here, the area is referred to as a virtual image display area), and the virtual image The display area is arranged so as to face the line of sight of the observer. That is, when viewed from the lateral direction of the vehicle, the angle between the virtual image display area where the virtual image is displayed and the road surface on which the vehicle runs is set to approximately 90 degrees (the virtual image display area is approximately 90 degrees from the road surface). are not arranged parallel to each other).

FIG. 11 shows the foreground viewed by an observer through the front windshield WS of the vehicle and a perspective image 1100 displayed in a virtual image display area 1000 overlapping the foreground while the vehicle is running. FIG. 4 is a diagram showing; FIG. 11 shows a perspective image expressed as if it were arranged parallel to the road surface using the linear perspective method. Here, the left line of the driving lane 1010 of the host vehicle is indicated by reference numeral 1011, the right line is indicated by reference numeral 1012, and the intersection of the extension of the left line 1011 and the extension of the right line 1012 is the vanishing point 1020. A perspective image 1100 in FIG. 11 is an arrow having a thickness 1120 in the height direction (Y-axis direction). The intersection point with 1102 is represented so as to coincide with the vanishing point 1020 . More specifically, the perspective image 1100 in FIG. 11 is expressed as if it is on the front side when viewed from the viewer, and has a first thickness portion 1111 having a first thickness 1121 in the height direction (Y-axis direction). , and a second thickness portion 1112 that is expressed to be deeper than the first thickness portion 1111 and has a second thickness 1122 in the height direction (Y-axis direction). In linear perspective (single-point perspective), the farther the object, the smaller (shorter) it is drawn, so the second thickness 1122 is shorter than the first thickness 1121 . In other words, in linear perspective (single-point perspective), the distance in the height direction (Y-axis direction) between the upper side 1101 passing through the upper part and the lower side 1102 passing through the lower part forming the thickness 1120 is gradually shortened as the distance increases. and zero at the vanishing point 1020. As a result, the perspective image 1100 can be represented pictorially as if it were arranged parallel to the plane of the driving lane 1010 .

However, as described above, the virtual image display area is not arranged substantially parallel to the road surface. , the distance between the far end 1300 of the perspective image 1100 and the driving lane 1010 is longer than the distance between the near end 1400 and the driving lane 1010, and the driving With respect to the plane parallel to the lane 1010, the far end 1300 of the perspective image 1100 is perceived as being raised toward the observer, making it difficult to perceive that it is parallel to the plane of the driving lane 1010. Therefore, improvement is required. There was room.

A summary of certain embodiments disclosed herein follows. It should be understood that these aspects are presented only to provide the reader with an overview of these particular embodiments and are not intended to limit the scope of this disclosure. Indeed, the present disclosure may encompass various aspects not described below.

An overview of the present disclosure relates to displaying an image that facilitates a sense of depth. More specifically, the present invention relates to providing a display control device, a head-up display device, a display control method, and the like that display an image that is easy to perceive as being parallel to the surface of the driving lane.

Therefore, the display control device, the head-up display device, the display control method, and the like described in this specification employ the following means in order to solve the above problems. This embodiment is a display control device that controls a head-up display device that allows an observer to visually recognize a virtual image of an image superimposed on the foreground. A thickness display process is performed to display as one or more perspective virtual images with gradually decreasing depth representation so as to converge, and the thickness vanishing point is positioned below the vanishing point of the foreground as seen from the observer. , is the gist of it.

Therefore, the display control device described in this specification is a display control device that controls a head-up display device that allows the observer to visually recognize a virtual image of an image superimposed on the foreground. and performs a thickness display process for displaying one or more perspective virtual images with a depth representation that gradually decreases so that the thickness converges on the vanishing point, and the thickness vanishing point is the foreground as seen from the observer is located below the vanishing point of

FIG. 1 is a diagram showing an application example of a vehicle virtual image display system to a vehicle. FIG. 2 is a diagram showing the configuration of the head-up display device. FIG. 3 is a diagram conceptually showing a virtual object placed at a target position in the real scene and an image displayed in the virtual image display area such that the virtual object is visually recognized at the target position in the real scene. FIG. 4 is a diagram showing an example of a foreground visually recognized by an observer and a virtual image of the first width display mode displayed superimposed on the foreground while the host vehicle is running. FIG. 5 is a diagram showing an example of a foreground visually recognized by an observer and a virtual image of the first width display mode superimposed on the foreground while the host vehicle is running. FIG. 6A is a diagram showing a modified example of a perspective image. FIG. 6B is a diagram showing a modification of the perspective image. FIG. 6C is a diagram showing a modification of the perspective image. FIG. 6D is a diagram showing a modification of the perspective image. FIG. 7 is a diagram showing an example of a foreground visually recognized by an observer and a virtual image of the first thickness display mode superimposed on the foreground while the host vehicle is running. FIG. 8A is a diagram showing a perspective virtual image V of a first thickness representation having a second vanishing point. FIG. 8B is a diagram showing a perspective virtual image V of a first thickness representation having a second vanishing point. FIG. 9 is a block diagram of a virtual image display system for vehicles according to some embodiments. FIG. 10 is a flowchart showing a display control method for executing display control processing for displaying a perspective virtual image in the first width display mode or the first thickness display mode. FIG. 11 is a diagram relating to a conventional example, and is a diagram showing an example of a foreground visually recognized by an observer and a virtual image displayed superimposed on the foreground while the host vehicle is running.

1 through 10 below provide a description of the configuration and operation of an exemplary vehicular virtual image display system. In addition, the present invention is not limited by the following embodiments (including the contents of the drawings). Of course, modifications (including deletion of constituent elements) can be added to the following embodiments. In addition, in the following description, descriptions of known technical matters are omitted as appropriate in order to facilitate understanding of the present invention.

Please refer to FIG. FIG. 1 is a diagram showing an application example of a vehicle virtual image display system to a vehicle. In FIG. 1, the left-right direction of a vehicle (an example of a moving object, hereinafter also referred to as the host vehicle) 1 (in other words, the width direction of the host vehicle 1) is the X axis (the positive direction of the X axis is the host vehicle). 1 when facing forward), and the vertical direction (in other words, the vehicle 1 height direction) is the Y-axis (the positive direction of the Y-axis is the upward direction), and the front-rear direction along a line segment perpendicular to each of the horizontal and vertical directions is the Z-axis (the positive direction of the Z-axis is the vertical direction). straight ahead direction of the vehicle 1). This point also applies to other drawings.

A head-up display device (an example of a virtual image display device) 20 in the vehicle virtual image display system 10 is a head-up display (HUD: Head-Up Display) provided in a dashboard 5 of a vehicle (an example of a moving object). ) device. The HUD device 20 emits the display light K toward the front windshield 2 (an example of a member to be projected), and the front windshield 2 directs the display light K of the image displayed by the HUD device 20 to the eyebox 200. reflect. The HUD device 20 collects not only information about the own vehicle 1 (hereinafter referred to as vehicle information) but also information other than the vehicle information in an integrated manner to the occupant of the own vehicle 1 (the occupant is typically the own vehicle 1). driver). The vehicle information includes not only the information of the own vehicle 1 itself but also the information of the outside of the own vehicle 1 related to the operation of the own vehicle 1 .

As illustrated, the vehicle virtual image display system 10 provided in the vehicle (self-vehicle) 1 is configured to display the positions of the left eye 700L and the right eye 700R of an observer (typically, the driver sitting in the driver's seat of the self-vehicle 1) and the line of sight. A vehicle exterior sensor composed of an eye position detection unit (line-of-sight detection unit) 409 for detecting a pupil (or face) for detecting a direction, a camera (for example, a stereo camera) for imaging the front (in a broad sense, the surroundings) of the own vehicle 1, and the like. 411 , a head-up display device (hereinafter also referred to as a HUD device) 20 and a display control device 30 that controls the HUD device 20 .

Further, the "eyebox" used in the description of the present embodiment means (1) an area in which the entire virtual image V of the image is visible, and at least a portion of the virtual image V of the image is not visible outside the area, and (2) At least a portion of the virtual image V of the image is visible within the region, and no portion of the virtual image V of the image is visible outside the region; outside the region, the entire virtual image V of the image is less than the predetermined brightness, or (4) when the HUD device 20 can display the virtual image V that can be stereoscopically viewed, at least part of the virtual image V is stereoscopic. It is an area where the virtual image V is visible, and a part of the virtual image V is not stereoscopically viewed outside the area. That is, when the observer places the eyes (both eyes) outside the eyebox 200, the observer cannot see the entire virtual image V of the image, or the visibility of the entire virtual image V of the image is very low and is difficult to perceive, or The virtual image V of the image cannot be viewed stereoscopically. The predetermined brightness is, for example, about 1/50 of the brightness of the virtual image of the image viewed at the center of the eyebox. The “eye box” is the same as the area (also called the eyelip) where the viewer's viewpoint position is assumed to be arranged in the vehicle in which the HUD device 20 is mounted, or a large portion (for example, 80% or more) of the eyelip. is set to include

The virtual image display area VS is a planar, curved, or partially curved area on which an image generated inside the HUD device 20 is formed as a virtual image V, and is also called an imaging plane. The virtual image display area VS is a position where a display surface 51 (for example, an exit surface of a liquid crystal light modulation element) of the display 40 (described later) of the HUD device 20 is formed as a virtual image by the virtual image optical system 90 described later. , the virtual image display area VS corresponds to the display surface 51 of the HUD device 20, which will be described later (in other words, the virtual image display area VS has a conjugate relationship with the display surface 51 of the display device 40, which will be described later). It can be said that the virtual image visually recognized in the display area VS corresponds to the image displayed on the display surface 51 of the HUD device 20, which will be described later. It is preferable that the virtual image display area VS itself has low visibility to the extent that it is not actually visible to the observer's eyes, or is difficult to be visually recognized.

In the virtual image display area VS, an angle (tilt angle θt in FIG. 1 ) formed between the horizontal direction (X-axis direction) and the horizontal direction (X-axis direction) of the vehicle 1, the center 205 of the eyebox 200, and the virtual image The angle formed by the line segment connecting the upper end VS1 of the display area VS and the line segment connecting the center of the eyebox and the lower end VS2 of the virtual image display area VS is defined as the vertical angle of view of the virtual image display area VS. An angle (vertical arrangement angle θd in FIG. 1) formed between the equisecting line and the horizontal direction (XZ plane) is set. The vertical arrangement angle θd is typically set downward from the horizontal direction (XZ plane), and is, for example, +2 degrees (here, the positive direction of the vertical arrangement angle θd is two degrees of the vertical angle of view). The bisector is the downward angle of depression from the center 205 of the eyebox 200, and the negative direction of the vertical arrangement angle θd is the elevation angle at which the bisector of the vertical angle of view is upward from the center 205 of the eyebox 200. can be said to be.).

In addition, the virtual image display area VS of the present embodiment has a tilt angle θt of approximately 90 [degrees] so that the virtual image display area VS faces approximately forward (positive direction of the Z axis). However, the tilt angle θt is not limited to this, and can be changed within the range of 30≦θt<90 [degree]. The tilt angle θt may be set to, for example, 60 [degrees], and the virtual image display area VS may be arranged such that the upper area is farther from the lower area as viewed from the observer. However, the range of the tilt angle θt is not limited to this.

FIG. 2 is a diagram showing one aspect of the configuration of the head-up display device. The HUD device 20 is installed, for example, in a dashboard (reference numeral 5 in FIG. 1). The HUD device 20 houses the display 40, the relay optical system 80, and the display 40 and the relay optical system 80, and has a light exit window through which the display light K from the display 40 can be emitted from the inside to the outside. a housing 22 having 21;

The display device 40 in FIG. 2 is typically mainly composed of a liquid crystal display panel (light modulation element) 50 and a light source unit 60 . The display surface 51 is the surface of the liquid crystal display panel 50 on the viewing side, and emits display light K for an image. By setting the angle of the display surface 51 with respect to the optical axis Kp of the display light K traveling from the center of the display surface 51 to the eyebox 200 (the center 205 of the eyebox 200) via the relay optical system 80 and the projection target portion, a virtual image is formed. The angle of the display area VS (including the tilt angle θt) can be set.

The liquid crystal display panel (light modulation element) 50 receives light from the light source unit 60 and emits display light K spatially modulated into a 2D image toward the relay optical system 80 (second mirror 82). The liquid crystal display panel 50 has, for example, a rectangular shape whose short side is the direction in which the pixels are arranged corresponding to the vertical direction (Y-axis direction) of the virtual image V viewed by the observer. An observer visually recognizes the light transmitted through the liquid crystal display panel 50 through the virtual image optical system 90 . The virtual image optical system 90 is a combination of the relay optical system 80 and the front windshield 2 shown in FIG. The light modulation element 50 (display device 40) may be a self-luminous display, or a projection display that projects an image onto a screen (for example, a reflective display panel using LCoS or DMD). but not limited to these. In this case, the display surface 51 is the screen of a projection display.

The light source unit 60 is composed of a light source (not shown) and an illumination optical system (not shown).

A light source (not shown) is, for example, a plurality of chip-type LEDs, and emits illumination light to a liquid crystal display panel (an example of a light modulation element) 22 . The light source unit 60 is composed of, for example, four light sources, which are arranged in a line along the long side of the liquid crystal display panel 50 . The light source unit 60 emits illumination light toward the liquid crystal display panel 50 under the control of the display control device 30 . The configuration of the light source unit 60 and the arrangement of the light sources are not limited to this.

The illumination optical system (not shown) includes, for example, one or more lenses (not shown) arranged in the emission direction of the illumination light from the light source unit 60, and a diffuser lens (not shown) arranged in the emission direction of the one or more lenses. a plate (not shown);

The light source unit 60 may be configured to be capable of local dimming, and the degree of illumination for each area of the display surface 51 may be changed under the control of the display control device 30 . Thereby, the light source unit 60 can adjust the brightness of the image displayed on the display surface 51 for each area.

The relay optical system 80 is arranged on the optical path of the display light K emitted from the display device 40 (light from the display device 40 toward the eyebox 200 ). consists of one or more optical elements that project onto the front windshield 2 of the The relay optical system 80 of FIG. 2 includes one concave first mirror 81 and one planar second mirror 82 .

The first mirror 81 has, for example, a free-form surface shape with positive optical power. In other words, the first mirror 81 may have a curved shape with different optical power for each region. can be different. Specifically, the first display light K11, the second display light K12, and the third display light K13 (see FIG. 2) traveling from each region of the display surface 51 toward the eyebox 200 are added by the relay optical system 80. The optical power may be different.

The second mirror 82 is, for example, a plane mirror, but is not limited to this, and may be a curved surface having optical power. That is, the relay optical system 80 is added according to the area (optical path) through which the display light K passes by synthesizing a plurality of mirrors (for example, the first mirror 81 and the second mirror 82 of this embodiment). Different optical powers may be used. Note that the second mirror 82 may be omitted. That is, the display light K emitted from the display device 40 may be reflected by the first mirror 81 to the projection target (front windshield) 2 .

Also, the relay optical system 80 includes two mirrors (reflecting optical system) in the present embodiment, but is not limited to this. Relay optics 80 may include one or more refractive optical members such as lenses, diffractive optical members such as holograms, reflective optical members different from those described above, or combinations thereof, which may include , may be added to the reflective optical system described above, and may replace part or all of the reflective optical system described above.

Further, the relay optical system 80 has a function of setting the distance to the virtual image display area VS and a function of generating a virtual image by enlarging the image displayed on the display surface 51, using this curved surface shape (an example of optical power). , but in addition to this, it may have a function of suppressing (correcting) the distortion of the virtual image that may occur due to the curved shape of the front windshield 2, or other known optical functions.

Relay optics 80 may also be rotatable with one or more actuators 86 , 87 attached thereto controlled by display controller 30 . For example, the actuator 86 may rotate (and/or move) the first mirror 81 about the first rotation axis AX1. The actuator 87 may rotate (and/or move) the second mirror 82 around the first rotation axis AX1.

The head-up display device 20 is controlled by a display control device 30, which will be described later. Near objects such as forks, road signs, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), and features (buildings, bridges, etc.), positions overlapping objects, or set based on objects By displaying an image at a position, it is possible to make an observer (typically, an observer seated in the driver's seat of the own vehicle 1) perceive visual augmented reality (AR). In the description of the present embodiment, an image whose displayed position can be changed according to a real object existing in the real scene or a target position in the vertical and horizontal directions and the depth in a predetermined space in the real scene determined by the display control device 30 is used as an AR image. An image is defined as an image, and an image whose display position is set regardless of the position of the real object is defined as a non-AR image. Further, in the description of the present embodiment, a perspective image is defined as an image composed of one or a plurality of images represented by a perspective drawing method that produces a sense of perspective, and an image that is not represented by a perspective drawing method is defined as a non-perspective image. . Perspective images are typically AR images.

The display control device 30, which will be described later, performs, for example, image rendering processing (graphic processing), display device driving processing, and the like, thereby controlling the aspect of the image V displayed by the HUD device 20 (perceived by the observer). can be done.

FIG. 3 is a diagram conceptually showing a virtual object placed at a target position in the real scene and an image displayed in the virtual image display area such that the virtual object is visually recognized at the target position in the real scene. As shown in FIG. 3, when viewed from the viewer 700, the depth direction is the Z-axis direction, the left-right direction (the width direction of the vehicle 1) is the X-axis direction, and the vertical direction (the vertical direction of the vehicle 1). is the Y-axis direction. Note that the direction away from the viewer is the positive direction of the Z axis, the leftward direction is the positive direction of the X axis, and the upward direction is the positive direction of the Y axis. direction.

The viewer 700 perceives the virtual object FU at a predetermined target position PT in the real scene by visually recognizing the virtual image V formed (imaged) in the virtual image display area VS through the projection target section 2. . A viewer visually recognizes the virtual image V of the image of the display light K reflected by the projection target section 2 . At this time, if the virtual image V is, for example, an arrow indicating a course, the arrow of the virtual image V is positioned in the virtual image display area VS so that the virtual object FU is placed at a predetermined target position PT in the foreground of the host vehicle 1 and visually recognized. to be displayed.

(Perspective virtual image V10 in first width display mode)
FIG. 4 is a diagram showing an example of a foreground visually recognized by an observer and a virtual image of the first width display mode superimposed on the foreground while the own vehicle 1 is running. A virtual image V10 in the first width display mode (hereinafter also referred to as a perspective virtual image V10) has a vanishing point VP10 (also referred to as a first vanishing point VP10, width vanishing point, or first width vanishing point). In other words, the figure of the perspective virtual image V10 has a shape having a vanishing point VP10 that gives a perception of depth. cross at

As shown in the figure, when the perspective virtual image V10 is arranged below the viewer (the position where the vertical arrangement angle θd is in the positive direction), the vanishing point VP10 is arranged above the position of the virtual image V10. be done. On the other hand, when the perspective virtual image V10 is arranged above the viewer (position where the vertical arrangement angle θd is in the negative direction), the vanishing point VP10 is arranged below the position of the virtual image.

Based on the positional relationship between the vanishing point VP10 and the perspective virtual image V10, the viewer obtains a sense of depth of the perspective virtual image V10 when viewed from the viewer. Since the perspective virtual image V10 has the vanishing point VP10 in this way, the viewer can easily estimate the depth position of the perspective virtual image V10.

On the other hand, as shown in FIG. 4, the driving lane 6 of the host vehicle 1 has a vanishing point VP0 (real vanishing point VP0). In this specific example, as seen from the viewer, there is a running lane 6 that runs straight ahead, and extension lines of boundary lines 7 and 8 on both sides of the running lane 6 substantially intersect (point becomes). Since the driving lane (road surface) 6 thus has the vanishing point VP0, the viewer obtains a sense of depth in the actual scene (driving lane 6).

In the vehicle virtual image display system 10 according to the present embodiment, as shown in FIG. 4, the vanishing point VP10 of the perspective virtual image V10 is closer to the viewer (from the observer than the vanishing point VP0 of the actual scene (driving lane 6)). When viewed, it is located below).

In general, when creating an image including a painting using linear perspective (perspective drawing), a vanishing point is used when arranging various objects in the image at their respective depth positions. For example, a straight line is virtually drawn radially from a predetermined vanishing point, and the outline of each object is made to follow the straight line, so that each object is perceived as fixed at a predetermined depth position. Although a plurality of vanishing points can be provided, here, for the sake of simplicity of explanation, a case where one vanishing point is provided in one image will be described.

When using a general linear perspective method (perspective drawing method), the extension lines of each boundary line forming the outline of the virtual object perceived by the observer intersect at the vanishing point VP0 of the foreground (running lane 6). The virtual image is displayed, that is, the virtual image is generated so that the position of the vanishing point VP10 of the virtual image coincides with the position of the vanishing point VP0 of the foreground (driving lane 6). The perspective virtual image V10 is generated so that the position of the vanishing point VP10 of V10 is closer to the viewer than the position of the vanishing point VP0 of the foreground (driving lane 6) (first width display processing).

This makes it easier for the observer to perceive that the perceived arrangement angle of the virtual object FU10 (the angle with respect to the road surface with the X2 axis in FIG. 3 as the rotation axis) is generally parallel to the driving lane 6. can.

The virtual object FU10 is an "arrow", and the target position PT at which the virtual object FU10 is placed (set) is usually made to match the height of the surface of the foreground (running lane 6) (that is, the set height to 0 m). However, this is an example and is not limited. In another example, the target position PT may be set at a position higher than the surface height l of the foreground (running lane 6) (that is, the set height may be set to 0.5 m or 1 m). In another example, the target position PT may be set at a position lower than the height of the surface of the foreground (running lane 6) (that is, the set height may be set to -1m or -2m). .

(Perspective virtual image V10 in first thickness display mode)
Also, the perspective virtual image V10 in the first thickness display mode has one or more thicknesses VT, and one or more thickness VT vanishing points VP20 (also referred to as second vanishing points or second width vanishing points). ). That is, the figure of the perspective virtual image V10 in the first thickness display mode has one or more thicknesses VT, and the farther the distance to be perceived by the viewer, the further it moves toward the second vanishing point VP20, and gradually One or more thicknesses VT are shortened. That is, the distance between the extension line V1C of the upper side forming the thickness VT and the extension line V1D of the lower side forming the thickness VT shown in FIG. The extension line V1C and the extension line V1D of the lower side intersect at the second vanishing point VP20).

As shown in FIG. 4, the perspective virtual image V10 in the first thickness display mode has two thicknesses VT1 and VT2. The perspective virtual image V10 in the first thickness display mode includes a first thickness portion VY1 parallel to the vertical and horizontal directions (XY plane), which is expressed as being closest to the viewer side of the trunk portion of the arrow, and the and a second thickness portion VY2 that is parallel to the vertical and horizontal directions (XY plane) and is expressed as if it is located closest to the viewer side of the head portion. The first thickness portion VY1 has a length (hereinafter also referred to as a first thickness) VT1 along the vertical direction (Y-axis direction) viewed from a viewer looking forward from the driver's seat. The thickness portion VY2 of 2 has a length (hereinafter also referred to as a second thickness) VT2 along the vertical direction (Y-axis direction) viewed from a viewer looking forward from the driver's seat. Here, the first thickness VT1 on the near side of the viewer is longer than the second thickness VT2 of the second thickness portion VY2 arranged farther than the first thickness portion VY1 having the first thickness VT1, A first thickness VT1 and a second thickness VT2 are set based on the upper side V1C and the lower side V1D converging on the second vanishing point VP20.

Based on how the thickness of the perspective virtual image V10 changes in the depth direction (relationship between the lengths of the first thickness VT1 and the second thickness VT2), the viewer determines the perspective virtual image V10 as seen from the viewer. get a sense of depth. In this way, the perspective virtual image V10 has the second vanishing point VP20 with respect to the thickness VT, making it easier for the viewer to estimate the depth position of the perspective virtual image V10.

In the virtual image display system 10 for a vehicle according to the second embodiment, as shown in FIG. 4, the second vanishing point VP20 of the thickness VT of the perspective virtual image V10 is higher than the vanishing point VP0 of the actual scene (driving lane 6). It is arranged on the near side (downward when viewed from the observer). That is, when a general linear perspective method (perspective drawing method) is used, the extension lines V1C and V1D of the boundary lines forming the thickness of the virtual object perceived by the observer correspond to the vanishing point VP0 of the foreground (running lane 6). That is, the virtual image is generated so that the position of the second vanishing point VP20 of the virtual image thickness VT matches the position of the vanishing point VP0 of the foreground (driving lane 6). However, in the second embodiment, the perspective virtual image V10 is arranged such that the position of the second vanishing point VP20 of the thickness VT of the perspective virtual image V10 is closer to the viewer than the position of the vanishing point VP0 of the foreground (running lane 6). V10 is generated (first thickness display processing).

In the example of FIG. 4, the first vanishing point VP10 in the width direction (X-axis direction) of the perspective virtual image V10 in the first width display process and the thickness direction (Y axial direction) coincide with the second vanishing point VP20, but not limited to this. The second vanishing point VP20 may be above the first vanishing point VP10 and below the actual scene vanishing point VP0 as seen from the observer. As a result, it is possible to display an image that is easy to perceive as being parallel to the surface of the driving lane while maintaining the thickness VT even at a distance (while suppressing a decrease in the thickness VT).

FIG. 5 is a diagram showing a modification of the perspective virtual image V10 in the first thickness display mode. The perspective virtual image V10 in the first thickness display mode shown in FIG. 2, but is not limited to this. The perspective virtual image V10 in the first thickness display mode has, as shown in FIG. The far end is a fourth thickness VT4, the extension line V1C of the upper side connecting the upper end of the third thickness VT3 and the upper end of the fourth thickness VT4, the lower end of the third thickness VT3 and the lower end of the fourth thickness VT4 are Even if the third thickness VT3 and the fourth thickness VT4 in one thickness portion VY3 are set so that the extension line V1D of the connecting lower side converges to the second vanishing point VP20 below the vanishing point VP0 of the actual scene. good.

6A, 6B, 6C, and 6D are diagrams showing modified examples of perspective images. Although the perspective virtual image V10 in the above embodiment is composed of one straight "arrow", it is not limited to this. The perspective virtual image V20 shown in FIG. 6A may be bent from the middle of the image (“arrow”). In this case, the left perspective line V1A can be defined as an extension of the left side until the perspective virtual image V20 bends, and the right perspective line V1B can be defined as an extension of the right side until the perspective virtual image V20 bends. can do.

A perspective virtual image V30 shown in FIG. 6B includes a first perspective virtual image V31 and a second perspective virtual image V32 arranged above the first perspective virtual image V31 and having a size smaller than that of the first perspective virtual image V31. Consists of In this case, the left perspective line V1A can be defined as a straight line connecting the left edge of the first perspective V31 and the left edge of the second perspective V32, and the right perspective line V1B can be defined as the right edge of the first perspective V31. and the right edge of the second perspective V32.

Also, the perspective virtual image V30 in the first thickness display mode shown in FIG. 6B has two thicknesses VT5 and VT6. The perspective virtual image V30 in the first thickness display mode includes a first thickness portion VY1 that is parallel to the vertical and horizontal directions (XY plane) of the first perspective virtual image V31 expressed as if it were on the near side of the viewer, and and a second thickness portion VY2 that is parallel to the vertical and horizontal directions (XY plane) of the second perspective virtual image V32 that is expressed to be farther than the first perspective virtual image V31. The first thickness portion VY1 has a first thickness VT1 along the vertical direction (Y-axis direction) as viewed by a viewer facing forward from the driver's seat. It has a second thickness VT2 along the vertical direction (Y-axis direction) viewed from the viewer facing the viewer. Here, the first thickness VT1 on the near side of the viewer is longer than the second thickness VT2 of the second thickness portion VY2 arranged farther than the first thickness portion VY1 having the first thickness VT1, A first thickness VT1 and a second thickness VT2 are set based on the upper side V1C and the lower side V1D converging on the second vanishing point VP20.

Also, the perspective virtual image V40 shown in FIG. 6C may be composed of a plurality of left and right divided perspective virtual images V41 to V49. In this case, the left perspective line V1A can be defined as a straight line (approximate straight line) connecting the left ends of the left perspectives V41 to V44, and the right perspective line V1B extends the right ends of the right perspectives V45 to V48. It can be defined as a connecting straight line (approximate straight line).

Also, the vanishing point VP10 (first vanishing point VP10) of the perspective virtual image V50 shown in FIG. 6D (the vanishing point VP0′ shown in FIG. 6D is located to the left of the vanishing point VP0). Even in such a case, the vanishing point VP10 (first vanishing point VP10) of the perspective virtual image V50 can be arranged closer to the viewer than the vanishing point VP0' (real vanishing point VP0).

(Perspective virtual image V60 in second width display mode)
FIG. 7 is a diagram showing an example of the foreground visually recognized by the observer and the virtual image of the second width display mode superimposed on the foreground while the own vehicle 1 is running. A virtual image V60 in the second width display mode (hereinafter also referred to as a perspective virtual image V60) is positioned farther than the first vanishing point VP10 of the perspective virtual image V10 in the first width display mode (the third vanishing point VP30). point VP30). That is, the figure of the perspective virtual image V60 has a shape having a third vanishing point VP30 that gives a perception of depth. 3 intersect at the vanishing point VP30.

As shown in the figure, when the perspective virtual image V60 in the second width display mode is arranged below the viewer (position where the vertical arrangement angle θd is in the positive direction), the third vanishing point VP30 is It is arranged above the position of the first vanishing point VP10. On the other hand, when the perspective virtual image V60 in the second width display mode is arranged above the viewer (at the position where the vertical arrangement angle θd is in the negative direction), the third vanishing point VP30 is the first vanishing point VP30. It is arranged below the position of the point VP10.

The HUD device 20 (display control device 30) of some embodiments sets the display mode of the perspective virtual image V to the first width display mode having the first vanishing point VP10, or The width display mode is changed to the second width display mode having the third vanishing point VP30 set farther than the first vanishing point VP10. The HUD device 20 (display control device 30) of some embodiments has a second width display mode having a third vanishing point VP30 if (10) the perspective virtual image V is a moving image that moves in the perspective direction or expands and contracts (an example of the second width display process), and (15) if the perspective virtual image V is a still image, a moving image that does not move in the perspective direction, or a moving image that does not expand or contract in the perspective direction, the first vanishing point VP10 is It is expressed in the first width display mode.

8A and 8B are diagrams showing a perspective virtual image V in a second width representation having a third vanishing point VP30. In the HUD device 20 (display control device 30) of some embodiments, the perspective virtual image V is (11) a moving image that is perceived as gradually expanding and contracting in the perspective direction (in the example of FIG. 8A, from the near side or (12) a moving image perceived as gradually moving in the perspective direction (in the example of FIG. 8B, In the case of a moving image represented by a perspective drawing that gradually moves toward the third vanishing point VP30 on the far side), it is represented in a second width display mode having the third vanishing point VP30 (second width display processing example). On the other hand, the HUD device 20 (display control device 30) determines that the perspective virtual image V is (16) a still image, (17) a moving image that is not perceived by stretching too much in the perspective direction, or (18) that is perceived by moving too much in the perspective direction. If it is a moving image that is not displayed, it is expressed in a first width display mode having a first vanishing point VP10 (an example of first width display processing). "(17) Moving image that does not expand or contract very much in the perspective direction or (18) Moving image that does not move much in the perspective direction" specifically includes, for example, a moving image that moves in the horizontal direction (X-axis direction), and a moving image that moves in the vertical direction (Y-axis direction). There is, in human perception, a moving image that moves only a minute distance below a predetermined threshold in a direction that can be perceived in the depth direction (Z-axis direction). A moving image that moves in the vertical direction (Y-axis direction) by a distance of 1/10 or less).

Further, the HUD device 20 (display control device 30) of some embodiments has a first vanishing point VP10 for the display mode of the perspective virtual image V according to the movement speed (deformation speed) of the moving image of the perspective virtual image V. The width display mode is changed to the first width display mode or the second width display mode having the third vanishing point VP30 set farther than the first vanishing point VP10. For example, the HUD device 20 (display control device 30) of some embodiments (20) displays a second width display having a third vanishing point VP30 if the moving speed (deformation speed) of the moving image of the perspective virtual image V is fast. (25) If the moving speed of the moving image of the perspective virtual image V is slow, it is represented in the first width display mode having the first vanishing point VP10 (first width display process). An example of width display processing).

In the HUD device 20 (display control device 30) of the above-described embodiment, the perspective virtual image V (21) gradually expands and contracts in the direction of perspective or moves gradually in the direction of perspective. , and if it is determined to be fast, it is expressed in a second width display mode having a third vanishing point VP30 (an example of second width display processing). On the other hand, the HUD device 20 (display control device 30) determines that the perspective virtual image V (26) gradually expands and contracts in the direction of perspective or moves gradually in the direction of perspective if the speed of the image is less than a predetermined image speed threshold. , if it is determined to be slow, it is expressed in a first width display mode having a first vanishing point VP10 (an example of first width display processing).

Further, the HUD device 20 (display control device 30) according to some embodiments changes the display mode of the perspective virtual image V according to the brightness in front (surroundings) of the host vehicle 1 to the above-described HUD device 20 having the first vanishing point VP10. Change to the first width display mode or the second width display mode having the third vanishing point VP30 set farther than the first vanishing point VP10 (an example of second width display processing). For example, the HUD device 20 (display control device 30) of some embodiments (30) displays a second width display having a third vanishing point VP30 when it is determined that the front (surroundings) of the own vehicle 1 is bright. (35) When it is determined that the front (surroundings) of the vehicle 1 is dark, the first width display mode having the first vanishing point VP10 is used (an example of first width display processing). .

Further, the HUD device 20 (display control device 30) of some embodiments sets the display mode of the perspective virtual image V in the vertical direction (Y-axis direction) of the perspective virtual image V, and in human perception, the depth direction (Z-axis direction). direction), the first width display mode having a first vanishing point VP10 or the second width display having a third vanishing point VP30 set farther than the first vanishing point VP10 Change to a mode. For example, the HUD device 20 (display control device 30) of some embodiments is (40) the length of the perspective virtual image V in the vertical direction (Y-axis direction), and in human perception, the depth direction (Z-axis direction) is equal to or longer than a predetermined threshold, the second width display mode having the third vanishing point VP30 (an example of second width display processing) is used, and (45) the perspective virtual image V It is the length in the direction (Y-axis direction), and when the distance in the depth direction (Z-axis direction) in human perception is shorter than a predetermined threshold, it is represented in the first width display mode having the first vanishing point VP10. (An example of first width display processing).

Further, the HUD device 20 (display control device 30) of some embodiments changes the display mode of the perspective virtual image V according to the length of the period in which the display is continued, to the above-described first vanishing point VP10. or the second width display mode having the third vanishing point VP30 set farther than the first vanishing point VP10. For example, the HUD device 20 (display control device 30) according to some embodiments, (50) when the period from the start of displaying the perspective virtual image V of the moving or still image to the end of displaying is equal to or longer than a predetermined threshold , expressed in a second width display mode having a third vanishing point VP30 (an example of second width display processing), and (55) a period from the start of displaying a perspective virtual image V of a moving or still image to the end of displaying it is shorter than a predetermined threshold, it is expressed in a first width display mode having a first vanishing point VP10 (an example of first width display processing).

Further, the HUD device 20 (display control device 30) according to some embodiments changes the display mode of the perspective virtual image V in the width direction from the first width display mode having the first vanishing point VP10 to the first vanishing point VP10. When changing to the second width display mode having the third vanishing point VP30 set farther than VP10, the display mode in the thickness direction of the perspective virtual image V is changed to the first width display mode having the second vanishing point VP20. The thickness display mode may be changed to the second width display mode having a fourth vanishing point (not shown) set farther than the second vanishing point VP20. Here, the first vanishing point VP10 and the second vanishing point VP20 may coincide, but are not limited to this. The second vanishing point VP20 may be above the first vanishing point VP10 and below the actual scene vanishing point VP0 as seen from the observer. Also, the third vanishing point (also referred to as the thickness vanishing point or the first thickness vanishing point) VP30 and the fourth vanishing point (not shown; also referred to as the second thickness vanishing point) may coincide. is not limited to The fourth vanishing point (not shown) may be above the third vanishing point VP30 and below the actual scene vanishing point VP0 as seen from the observer.

FIG. 9 is a block diagram of a vehicular virtual image display system, according to some embodiments. The display controller 30 comprises one or more I/O interfaces 31 , one or more processors 33 , one or more image processing circuits 35 and one or more memories 37 . Various functional blocks illustrated in FIG. 9 may be implemented in hardware, software, or a combination of both. FIG. 9 is only one embodiment and the illustrated components may be combined into fewer components or there may be additional components. For example, image processing circuitry 35 (eg, a graphics processing unit) may be included in one or more processors 33 .

As shown, processor 33 and image processing circuitry 35 are operatively coupled with memory 37 . More specifically, the processor 33 and the image processing circuit 35 execute a program stored in the memory 37 to generate and/or transmit image data, for example, to display the vehicle virtual image display system 10 (display 40) can be controlled. Processor 33 and/or image processing circuitry 35 may include at least one general purpose microprocessor (e.g., central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA). , or any combination thereof. Memory 37 includes any type of magnetic media such as hard disks, any type of optical media such as CDs and DVDs, any type of semiconductor memory such as volatile memory, and non-volatile memory. Volatile memory may include DRAM and SRAM, and non-volatile memory may include ROM and NVRAM.

As shown, processor 33 is operatively coupled with I/O interface 31 . The I/O interface 31 communicates (CAN communication). Note that the communication standard adopted by the I/O interface 31 is not limited to CAN, and includes, for example, CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport : MOST is a registered trademark), a wired communication interface such as UART or USB, or a personal area network (PAN) such as a Bluetooth network, a local network such as an 802.11x Wi-Fi network. It includes an in-vehicle communication (internal communication) interface, which is a short-range wireless communication interface within several tens of meters such as an area network (LAN). In addition, the I / O interface 31 is a wireless wide area network (WWAN0, IEEE802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE802.16e base (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, An external communication (external communication) interface such as a wide area communication network (for example, Internet communication network) may be included according to a cellular communication standard such as 5G.

As illustrated, the processor 33 is interoperably connected to the I/O interface 31, thereby communicating with various other electronic devices connected to the vehicle virtual image display system 10 (I/O interface 31). Information can be exchanged. The I/O interface 31 includes, for example, a vehicle ECU 401, a road information database 403, a vehicle position detection unit 405, an operation detection unit 407, an eye position detection unit 409, an external sensor 411, a brightness detection unit 413, an IMU 415, portable information A terminal 417, an external communication device 419, etc. are operatively coupled. Note that the I/O interface 31 may include a function of processing (converting, calculating, and analyzing) information received from other electronic devices or the like connected to the vehicle virtual image display system 10 .

Display 40 is operatively coupled to processor 33 and image processing circuitry 35 . Accordingly, the image displayed by light modulating element 50 may be based on image data received from processor 33 and/or image processing circuitry 35 . The processor 33 and image processing circuit 35 control the image displayed by the light modulation element 50 based on information obtained from the I/O interface 31 .

The vehicle ECU 401 detects the state of the vehicle 1 (for example, travel distance, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine Rotation speed, motor rotation speed, steering angle, shift position, drive mode, various warning states, posture (including roll angle and/or pitch angle), vehicle vibration (vibration magnitude, frequency, and/or frequency )), etc., collects and manages (may also include control) the state of the vehicle 1, and as part of the function, the numerical value of the state of the vehicle 1 (for example, A signal indicating the vehicle speed of the host vehicle 1 can be output to the processor 33 of the display control device 30 . The vehicle ECU 401 simply transmits a numerical value detected by a sensor or the like (for example, the pitching angle is 3 [degrees] in the forward tilting direction) to the processor 33, or alternatively, transmits the numerical value detected by the sensor. determination result based on one or more states of the own vehicle 1 including, for example, that the own vehicle 1 satisfies a predetermined forward lean condition, or/and an analysis result (for example, brake pedal open In combination with the degree information, the fact that the vehicle is leaning forward due to braking) may be transmitted to the processor 33 . For example, the vehicle ECU 401 may output to the display control device 30 a signal indicating a determination result that the host vehicle 1 satisfies a predetermined condition stored in advance in a memory (not shown) of the vehicle ECU 401 . Note that the I/O interface 31 may acquire the above-described information from sensors and switches provided in the own vehicle 1 without using the vehicle ECU 401 .

In addition, the vehicle ECU 401 may output to the display control device 30 an instruction signal for instructing an image displayed by the virtual image display system 10 for a vehicle. Necessity-related information that serves as a basis for determining the necessity and/or the notification necessity may be added to the instruction signal and transmitted.

The road information database 403 is included in a navigation device (not shown) provided in the own vehicle 1 or an external server connected to the own vehicle 1 via an external communication interface (I/O interface 31). Based on the position of the vehicle 1 acquired from the position detection unit 405, road information (lanes, white lines, stop line, pedestrian crossing, road width, number of lanes, intersection, curve, branch road, traffic regulation, etc.), presence/absence of feature information (buildings, bridges, rivers, etc.), position (including distance to own vehicle 1) ), direction, shape, type, detailed information, etc. may be read and sent to the processor 33 . The road information database 403 may also calculate an appropriate route (navigation information) from the departure point to the destination and output to the processor 33 a signal indicating the navigation information or image data indicating the route.

The vehicle position detection unit 405 is a GNSS (global navigation satellite system) or the like provided in the vehicle 1, detects the current position and direction of the vehicle 1, and sends a signal indicating the detection result to the processor 33. The information is output to the road information database 403 , a portable information terminal 417 and/or an external communication device 419 to be described later, via or directly. The road information database 403, a portable information terminal 417 (to be described later), and/or an external communication device 419 acquire position information of the vehicle 1 from the vehicle position detection unit 405 continuously, intermittently, or at each predetermined event. , information about the surroundings of the own vehicle 1 may be selected/generated and output to the processor 33 .

The operation detection unit 407 is, for example, a CID (Center Information Display) of the vehicle 1, a hardware switch provided on an instrument panel, or a software switch combining an image and a touch sensor. Operation information based on an operation by one passenger (a user sitting in the driver's seat and/or a user sitting in the passenger's seat) is output to the processor 33 . For example, the operation detection unit 407 sets the display area setting information based on the operation of moving the virtual image display area 100, the eye box setting information based on the operation of moving the eye box 200, and the observer's eye position 700 by the user's operation. Information based on the operation to be performed is output to the processor 33 .

The eye position detection unit 409 includes a camera such as an infrared camera that detects the eye position 700 (see FIG. 1) of the observer sitting in the driver's seat of the vehicle 1, and outputs the captured image to the processor 33. good too. The processor 33 acquires a captured image (an example of information that can estimate the eye position 700) from the eye position detection unit 409, and analyzes the captured image by a method such as pattern matching to determine the eye positions of the observer. The coordinates of 700 may be detected and a signal indicative of the detected coordinates of eye position 700 may be output to processor 33 .

In addition, the eye position detection unit 409 obtains an analysis result obtained by analyzing the image captured by the camera (for example, where the observer's eye position 700 belongs in a spatial region corresponding to a plurality of preset display parameters). ) may be output to the processor 33 . Note that the method of acquiring the eye position 700 of the observer of the own vehicle 1 or the information that can estimate the eye position 700 of the observer is not limited to these, and a known eye position detection (estimation) technique can be used. may be obtained using

Further, the eye position detection unit 409 detects the moving speed and/or moving direction of the observer's eye position 700, and outputs a signal indicating the moving speed and/or moving direction of the observer's eye position 700 to the processor 33. can be output to

Further, the eye position detection unit 409 outputs (10) a signal indicating that the observer's eye position 700 is outside the eyebox 200, and (20) the observer's eye position 700 is estimated to be outside the eyebox 200. When a signal or (30) a signal predicted that the observer's eye position 700 is outside the eyebox 200, it is determined that a predetermined condition is satisfied, and a signal indicating the state is output to the processor 33. may

(20) The signal that the observer's eye position 700 is estimated to be outside the eyebox 200 includes (21) a signal indicating that the observer's eye position 700 cannot be detected, and (22) the observer's eye position 700. After the movement is detected, a signal indicating that the observer's eye position 700 cannot be detected; includes, for example, a signal indicating that it is within a predetermined coordinate from the boundary 200A).

(30) The signal predicted that the observer's eye position 700 is outside the eye box 200 is (31) the newly detected eye position 700 is stored in advance in the memory 37 with respect to the previously detected eye position 700. (32) eye position movement speed is stored in memory 37 in advance. and a signal indicating that the eye position moving speed is equal to or higher than the eye position moving speed threshold.

Also, the eye position detection unit 409 may have a function as the line-of-sight direction detection unit 409 . The line-of-sight direction detection unit 409 may include an infrared camera or a visible light camera that captures the face of an observer sitting in the driver's seat of the vehicle 1 , and may output the captured image to the processor 33 . The processor 33 acquires a captured image (an example of information for estimating the line-of-sight direction) from the line-of-sight direction detection unit 409, and analyzes the captured image to determine the line-of-sight direction (and/or the gaze position) of the observer. can be specified. Note that the line-of-sight direction detection unit 409 may analyze the captured image from the camera and output to the processor 33 a signal indicating the line-of-sight direction (and/or the gaze position) of the observer, which is the analysis result. It should be noted that the method of acquiring information that can estimate the line-of-sight direction of the observer of the own vehicle 1 is not limited to these methods, and includes an EOG (Electro-oculogram) method, a corneal reflection method, a scleral reflection method, a Purkinje image It may also be obtained using other known gaze direction detection (estimation) techniques such as detection methods, search coil methods, infrared fundus camera methods.

The vehicle exterior sensor 411 detects real objects existing around the vehicle 1 (front, side, and rear). Real objects detected by the sensor 411 outside the vehicle include, for example, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), road surfaces of driving lanes, lane markings, roadside objects, and/or features (buildings, etc.), which will be described later. and so on. Exterior sensors include, for example, radar sensors such as millimeter wave radar, ultrasonic radar, and laser radar, cameras, or a detection unit consisting of a combination thereof, and processing detection data from the one or more detection units ( data fusion) and a processing device. A conventional well-known method is applied to object detection by these radar sensors and camera sensors. By object detection by these sensors, the presence or absence of a real object in the three-dimensional space, and if the real object exists, the position of the real object (relative distance from the own vehicle 1, the traveling direction of the own vehicle 1) Horizontal position, vertical position, etc.), size (horizontal direction (horizontal direction), height direction (vertical direction), etc.), movement direction (horizontal direction (horizontal direction) , depth direction (front-rear direction)), movement speed (horizontal direction (left-right direction), depth direction (front-rear direction)), and/or type and the like may be detected. One or a plurality of vehicle exterior sensors 411 detects a real object in front of the own vehicle 1 at each detection cycle of each sensor, and obtains real object information (presence or absence of a real object, whether or not a real object exists), which is an example of real object information. Information such as the position, size and/or type of each real object, if any, can be output to the processor 33 . Note that the real object information may be transmitted to the processor 33 via another device (for example, the vehicle ECU 401). Also, when using a camera as a sensor, an infrared camera or a near-infrared camera is desirable so that a real object can be detected even when the surroundings are dark, such as at night. Moreover, when using a camera as a sensor, a stereo camera that can acquire distance and the like by parallax is desirable.

The brightness detection unit 413 converts the illuminance or luminance of a predetermined range of the foreground that exists in front of the passenger compartment of the vehicle 1 into the outside world brightness (an example of brightness information), or converts the illuminance or luminance in the passenger compartment into the vehicle interior brightness ( (an example of brightness information). The brightness detection unit 413 is, for example, a phototransistor or a photodiode, and is mounted on the instrument panel, the room mirror, the HUD device 20, or the like of the own vehicle 1 shown in FIG.

The IMU 415 includes one or more sensors (e.g., accelerometers and gyroscopes) configured to detect the position, orientation, and changes thereof (velocity of change, acceleration of change) of the host vehicle 1 based on inertial acceleration. ). The IMU 415 analyzes the detected values (the detected values include signals indicating the position and orientation of the host vehicle 1 and their changes (change speed and change acceleration)), the results of analyzing the detected values, It may be output to processor 33 . The analysis result is a signal or the like indicating whether or not the detected value satisfies a predetermined condition. Therefore, the signal may be a signal indicating that the behavior (vibration) of the own vehicle 1 is small.

The mobile information terminal 417 is a smart phone, a notebook computer, a smart watch, or other information equipment that can be carried by the observer (or other occupant of the own vehicle 1). By pairing with the mobile information terminal 417, the I/O interface 31 can communicate with the mobile information terminal 417, and can read data recorded in the mobile information terminal 417 (or a server through the mobile information terminal). to get The mobile information terminal 417 has, for example, the same functions as the road information database 403 and the own vehicle position detection unit 405 described above, acquires the road information (an example of real object related information), and transmits it to the processor 33. may Also, the mobile information terminal 417 may acquire commercial information (an example of real object related information) related to commercial facilities near the own vehicle 1 and transmit it to the processor 33 . In addition, the mobile information terminal 417 transmits schedule information of the owner of the mobile information terminal 417 (for example, an observer), incoming call information at the mobile information terminal 417, mail reception information, etc. to the processor 33, and the processor 33 and the image Processing circuitry 35 may generate and/or transmit image data for these.

The external communication device 419 is a communication device that exchanges information with the own vehicle 1, for example, other vehicles connected to the own vehicle 1 by vehicle-to-vehicle communication (V2V: vehicle-to-vehicle), vehicle-to-pedestrian communication (V2P: vehicle Pedestrian) connected by pedestrians (mobile information terminals carried by pedestrians), road-to-vehicle communication (V2I: Vehicle To roadside Infrastructure) is a network communication device connected by, in a broad sense, the vehicle 1 Includes everything connected by communication (V2X: Vehicle To Everything). The external communication device 419 acquires, for example, the positions of pedestrians, bicycles, motorcycles, other vehicles (preceding vehicles, etc.), road surfaces, lane markings, roadside objects, and/or features (buildings, etc.), and sends them to the processor 33. may be output. Further, the external communication device 419 has the same function as the vehicle position detection unit 405 described above, may acquire the position information of the vehicle 1 and transmit it to the processor 33, and furthermore, the road information database 403 described above may be used. and may acquire the road information (an example of real object related information) and transmit it to the processor 33 . Information acquired from the external communication device 419 is not limited to the above.

The software components stored in memory 37 include display parameter setting module 502 , graphics module 504 , light source driving module 506 and actuator driving module 508 .

FIG. 10 shows the perspective virtual image in the first display mode (first width display mode, first thickness display mode) or second display mode (second width display mode, second thickness display mode). FIG. 4 is a flow diagram illustrating a method S100 of performing display control processing to cause the display to be displayed. The method S100 is performed in a HUD device 20 that includes a display and a display controller 30 that controls the HUD device 20 . Some acts in method S100 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

The display control device 30 (a graphic module 504 to be described later) receives, for example, an appropriate route (navigation information) from the departure point to the destination from the road information database 403, real object information (presence or absence of a real object, real object exists, information such as the position, size and/or type of each real object) is acquired, and the virtual object to be displayed is selected and set (S110).

The display parameter setting module 502 changes the display mode of the perspective virtual image V based on one or more pieces of information acquired from the I/O interface 31, as shown in S131. The display parameter setting module 502, based on one or more pieces of information acquired from the I/O interface 31, (11) when setting the perspective virtual image V to a moving image that moves in the perspective direction or expands and contracts, the third vanishing point The second width display mode having VP30 is set (an example of second width display processing), and (12) the perspective virtual image V is changed to a still image, a moving image that does not move much in the perspective direction, or a moving image that does not expand or contract in the perspective direction. , the first width display mode having the first vanishing point VP10 is set (an example of first width display processing). In some embodiments, the display parameter setting module 502 also (16) moves or scales the perspective virtual image V in the perspective direction based on one or more information obtained from the I/O interface 31. (17) Perspective virtual image V is set to a still image with a relatively far perspective When setting a moving image that does not move in any direction or a moving image that does not expand or contract in the perspective direction, the first thickness display mode having the second vanishing point VP20 is set (an example of first thickness display processing).

In addition, as shown in S132, the display parameter setting module 502 (21) sets the moving speed of the moving image of the perspective virtual image V faster than a predetermined threshold based on one or more pieces of information acquired from the I/O interface 31. In this case, the second width display mode having the third vanishing point VP30 is set (an example of the second width display process), and (22) the moving speed of the perspective virtual image V is set faster than a predetermined threshold. , a first width display mode having a first vanishing point VP10 (an example of first width display processing). Further, in some embodiments, the display parameter setting module 502, as shown in S132, based on one or more pieces of information obtained from the I/O interface 31, (26) moving speed of the moving image of the perspective virtual image V is set faster than a predetermined threshold value, a second thickness display mode having a fourth vanishing point (not shown) is set (an example of second thickness display processing), and (27) movement of the moving image of the perspective virtual image V When the speed is set faster than the predetermined threshold, the first thickness display mode having the second vanishing point VP20 is set (an example of first thickness display processing).

Further, as shown in S133, the display parameter setting module 502, based on the information (but not limited to this) acquired from the brightness detection unit 413, (31) determines that the front (surroundings) of the vehicle 1 is bright. If it is determined, the perspective virtual image V is set to the second width display mode having the third vanishing point VP30 (an example of the second width display processing), and (32) the front (surroundings) of the vehicle 1 is dark. , the perspective virtual image V is set to the first width display mode having the first vanishing point VP10 (an example of first width display processing). Further, in some embodiments, the display parameter setting module 502, as shown in S133, based on information (but not limited to) obtained from the brightness detection unit 413, (36) If it is determined that the front (surroundings) is bright, the perspective virtual image V is set to the second thickness display mode having a fourth vanishing point (not shown) (an example of second thickness display processing), (37) When it is determined that the front (surroundings) of the vehicle 1 is dark, the perspective virtual image V is set to the first thickness display mode having the second vanishing point VP20 (an example of first thickness display processing).

In addition, as shown in S134, the display parameter setting module 502 acquires road information acquired from the road information database 403 and position information of a real object existing in the foreground acquired from the vehicle exterior sensor 411 (but not limited to this). ), (41) is the length in the vertical direction (Y-axis direction) of the perspective virtual image V pointing to the guide route or the real object existing in the foreground. , is equal to or longer than a predetermined threshold stored in advance in the memory 37, the perspective virtual image V is set to a second width display mode having a third vanishing point VP30 (an example of second width display processing). , (42) is the length of the perspective virtual image V in the vertical direction (Y-axis direction), and when the distance in the depth direction (Z-axis direction) in human perception is shorter than a predetermined threshold, the perspective virtual image V is placed in the first A first width display mode having a vanishing point VP10 is set (an example of first width display processing). Further, in some embodiments, the display parameter setting module 502, as shown in S134, uses the road information acquired from the road information database 403 and the position information ( (46) is the vertical direction (Y-axis direction) length of the perspective virtual image V indicating the real object existing in the guide route or the foreground. Z-axis direction) is equal to or longer than a predetermined threshold pre-stored in the memory 37, the perspective virtual image V is set to the second thickness display mode having a fourth vanishing point (not shown). (Example of Second Thickness Display Processing), (47) The length of the perspective virtual image V in the vertical direction (Y-axis direction). If it is short, the perspective virtual image V is set to the first thickness display mode having the second vanishing point VP20 (an example of first thickness display processing).

In addition, as shown in S135, the display parameter setting module 502, based on one or more information acquired from the I/O interface 31, (51) displays the moving or still image perspective virtual image V after starting to display it. If the period until the end is equal to or longer than a predetermined threshold value predetermined in the memory 37, the perspective virtual image V is set to the second width display mode having the third vanishing point VP30 (second width display processing (Example), (52) When the period from the start of displaying the perspective virtual image V of the moving or still image to the end of displaying it is shorter than a predetermined threshold stored in advance in the memory 37, the perspective virtual image V is displayed at the first vanishing point. A first width display mode having VP10 is set (an example of first width display processing). In some embodiments, the display parameter setting module 502 may also (56) generate a moving or still perspective virtual image V based on one or more information obtained from the I/O interface 31, as shown at S135. is equal to or longer than a predetermined threshold value predetermined in the memory 37, the perspective virtual image V is displayed in a second thickness display mode having a fourth vanishing point (not shown). (an example of second thickness display processing), and (57) when the period from the start of displaying the perspective virtual image V of the moving or still image to the end of displaying is shorter than a predetermined threshold stored in advance in the memory 37 , the perspective virtual image V is set to a first thickness display mode having a second vanishing point VP20 (an example of first thickness display processing).

Graphics module 504 includes various known software components for performing image processing, such as rendering, to generate image data, and to drive display 40 . The graphic module 504 also controls the type (moving image, still image, shape), arrangement (positional coordinates, angle), size, display distance (in the case of 3D), visual effect (for example, brightness, (transparency, saturation, contrast, or other visual properties) may be included. The graphic module 504 stores the type of image (one example of the display parameter), the positional coordinates of the image (one example of the display parameter), the angle of the image (the pitching angle with the X direction as the axis, and the Y direction as the axis). yaw rate angle around the axis, rolling angle around the Z direction, etc., which are examples of display parameters), image size (one example of display parameters), image color (hue, saturation , one example of display parameters set by brightness, etc.), and the intensity of perspective expression of an image (one of display parameters set by the position of a vanishing point, etc.) to make an image visible to the observer. Data may be generated to drive the light modulating element 50 .

Light source driving module 506 includes various known software components for performing driving of light source unit 24 . The light source driving module 506 can drive the light source unit 24 based on the set display parameters.

Actuator drive module 508 includes various known software components for performing driving first actuator 28 and/or second actuator 29. One actuator 28 and a second actuator 29 can be driven.

The eye position detection module 510 detects the eye position 700 of the observer. The eye position detection module 510 detects coordinates indicating the observer's eye position 700 (positions in the X and Y axis directions and is an example of a signal indicating the eye position 700), coordinates indicating the observer's eye height ( , which is an example of a signal indicating the eye position 700), and the coordinates indicating the position in the eye height and depth direction of the observer (which are the positions in the Y and Z axis directions). , which is an example of a signal indicating the eye position 700), and/or coordinates indicating the eye position 700 of the observer (which are positions in the X, Y, and Z axis directions and are signals indicating the eye position 700). is an example.) includes various software components for performing various operations related to detecting.

Note that the positions of the first vanishing point VP10, the second vanishing point VP20, the third vanishing point VP30, and the fourth vanishing point (not shown) are the observer's eye position 700 detected by the eye position detection unit 409, or The attitude of the own vehicle 1 detected by the IMU 415 may be used for adjustment. For example, the display control device 30 (display parameter setting module 502) changes the position of the vehicle 1 (the pitching angle (the angle around the X direction)) by changing the first vanishing point VP10, the second vanishing point VP20, The positions of the third vanishing point VP30 and the fourth vanishing point (not shown) can be adjusted. When the pitching angle is such that the front of the vehicle 1 is lowered, the display control device 30 (display parameter setting module 502) controls the first vanishing point VP10, the second vanishing point VP20, and the virtual image display area VS as seen from the observer. When the relative positions of the third vanishing point VP30 and the fourth vanishing point (not shown) are adjusted to be on the upper side, and the pitching angle is such that the front of the own vehicle 1 rises, a virtual image seen from the observer The relative positions of the first vanishing point VP10, the second vanishing point VP20, the third vanishing point VP30, and the fourth vanishing point (not shown) with respect to the display area VS can be adjusted to be on the lower side.

The operations of the processing processes described above may be implemented by executing one or more functional modules of an information processing device such as a general purpose processor or an application specific chip. These modules, combinations of these modules, and/or combinations with known hardware that can replace their functions are all within the scope of protection of the present invention.

The functional blocks of the vehicular virtual image display system 10 are optionally implemented in hardware, software, or a combination of hardware and software to carry out the principles of the various described embodiments. 9 may optionally be combined or separated into two or more sub-blocks to implement the principles of the described embodiments; will be understood by those skilled in the art. Accordingly, the description herein optionally supports any possible combination or division of the functional blocks described herein.

1: Own vehicle 2: Projected member 5: Dashboard 6: Road surface 6: Driving lane (road surface)
7: Boundary line 8: Boundary line 10: Vehicle virtual image display system 20: HUD device (head-up display device)
21 : light exit window 22 : housing 24 : light source unit 28 : first actuator 29 : second actuator 30 : display control device 31 : I/O interface 33 : processor 35 : image processing circuit 37 : memory 40 : display 50 : Light modulation element 51 : Display surface 60 : Light source unit 80 : Relay optical system 81 : First mirror 82 : Second mirror 86 : Actuator 87 : Actuator 90 : Virtual image optical system 100 : Virtual image display area 200 : Eye box 200A : Boundary 205: Center 401: Vehicle ECU
403: road information database 405: own vehicle position detection unit 407: operation detection unit 409: eye position detection unit (line-of-sight direction detection unit)
411 : Exterior sensor 413 : Brightness detection unit 417 : Personal digital assistant 419 : External communication device 502 : Display parameter setting module 504 : Graphic module 506 : Light source drive module 508 : Actuator drive module 510 : Eye position detection module 700 : Observation Person 700L: Eye position (left eye)
700R: Eye position (right eye)
1000 : Virtual image display area 1010 : Driving lane 1011 : Left line 1012 : Right line 1020 : Vanishing point 1100 : Perspective image 1101 : Upper side 1102 : Lower side 1111 : First thickness portion 1112 : Second thickness portion 1120 : Thickness 1121 : First thickness 1122: Second thickness 1300: Far end 1400: Near end AX1: First rotation axis FU: Virtual object FU10: Virtual object K: Display light K11: First display light K12: Second display light K13 : Third display light Kp : Optical axis PT : Target position V : Image V : Virtual image V10 : Perspective virtual image V1A : Extension line (left perspective line)
V1B: Extension line (right perspective line)
V1C: Extension line (upper side)
V1D: Extension line (lower side)
V20: perspective virtual image VP0: real scene vanishing point VP0: vanishing point VP0': vanishing point VP10: first vanishing point (width vanishing point, first width vanishing point)
VP20: Second vanishing point (thickness vanishing point, first thickness vanishing point)
VP30: Third vanishing point (second width vanishing point)
VS: virtual image display area VS1: upper end VS2: lower end VT: thickness VT1: first thickness VT2: second thickness VT3: third thickness VT4: fourth thickness VT5: thickness VT6: thickness VY1: first thickness Part VY2: Second thickness part VY3: Thickness part WS: Front windshield θd: Vertical arrangement angle θt: Tilt angle

Claims (12)

A display control device (30) for executing display control in a head-up display device that allows an observer to view a virtual image of the image superimposed on the foreground by projecting the image onto a projection target member,
one or more processors (33);
a memory (37);
one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33);
Said processor (33)
One or more perspective virtual images having a thickness (VT) in the vertical direction as seen from the observer, and with a depth expression that gradually decreases so that the thickness (VT) converges to a thickness vanishing point (VP20). Execute thickness display processing to display as
The thickness vanishing point (VP20) is positioned below the foreground vanishing point (VP0) as seen from the observer.
A display controller (30). The one or more perspective virtual images are represented to be a first thickness portion (VY1) visually recognized by an observer and to be farther than the first thickness portion (VY1), and the first thickness the second thickness portion (VY2) shorter than the portion (VY1);
The thickness vanishing point (VP20) consists of an extension line (V1C) connecting a predetermined upper end of the first thick portion (VY1) and a predetermined upper end of the second thick portion (VY2), and A point where an extension line (V1D) of a line segment connecting a prescribed lower end of the first thick portion (VY1) and a prescribed lower end of the second thick portion (VY2) intersects,
A display control device (30) according to claim 1. Said processor (33)
Width display processing for displaying one or a plurality of perspective virtual images having a width in the horizontal direction as viewed from the observer and with a depth expression that gradually decreases so that the width converges to a width vanishing point (VP10). and run
The width vanishing point (VP10) is arranged below the foreground vanishing point (VP0) as seen from the observer,
In the thickness display process, the thickness vanishing point (VP20) is arranged above the width vanishing point (V10) and below the foreground vanishing point (VP0) as seen from the observer.
A display control device (30) according to claim 1. Said processor (33)
a first thickness representation displayed as said one or more perspective virtual images in a first thickness representation mode having a first thickness vanishing point positioned below said foreground vanishing point (VP0) as viewed by an observer. processing;
A second thickness representation displayed as the one or more perspective virtual images in a second thickness representation mode having a second thickness vanishing point located above the first thickness vanishing point as viewed by an observer. perform the processing and
A display control device (30) according to claim 1. Said processor (33)
if the perspective virtual image is a still image, executing the first thickness display process;
If the perspective virtual image is a moving image, executing the second thickness display process.
A display control device (30) according to claim 4. Said processor (33)
If the perspective virtual image is a moving image that does not expand or contract in the perspective direction or a moving image that does not move in the perspective direction when viewed from the observer, executing the first thickness display process,
If the perspective virtual image is a moving image that is perceived by an observer to expand and contract in the perspective direction or a moving image that is perceived to be moving in the perspective direction, the second thickness display processing is performed. run the
A display control device (30) according to claim 4. Said processor (33)
When the surroundings are dark, executing the first thickness display process,
When the surroundings are bright, executing the second thickness display process;
A display control device (30) according to claim 4. Said processor (33)
if the moving speed or deformation speed of the perspective virtual image is slow, executing the first thickness display process;
When the moving speed or deformation speed of the perspective virtual image is fast, executing the second thickness display process;
A display control device (30) according to claim 4. Said processor (33)
if the vertical length of the perspective virtual image is shorter than a predetermined threshold, executing the first thickness display process;
When the vertical length of the perspective virtual image is longer than the predetermined threshold, the second thickness display process is executed.
A display control device (30) according to claim 4. Said processor (33)
if the period from the start of displaying the perspective to the end of displaying the perspective is shorter than a predetermined threshold, executing the first thickness display process;
If the period from the start of displaying the perspective to the end of displaying the perspective is longer than the predetermined threshold, executing the second thickness display process;
A display control device (30) according to claim 4. A display control device (30) according to any one of claims 1 to 10;
a light modulation element (50) for emitting display light;
A head-up display device (20), comprising: a relay optical system (80) for directing the display light from the light modulation element (50) to a projection target. An image display control method comprising:
One or more perspective virtual images having a thickness (VT) in the vertical direction as seen from the observer, and with a depth expression that gradually decreases so that the thickness (VT) converges to a thickness vanishing point (VP20). Execute thickness display processing to display as
The thickness vanishing point (VP20) is positioned below the foreground vanishing point (VP0) as seen from the observer.
Display control method.

Images