JP7456290B2 - heads up display device - Google Patents

Description

The present invention relates to a head-up display (HUD) device etc. that displays a virtual image, and for example, to a HUD device etc. having a light field type stereoscopic display section (light field display).

Patent Document 1 shows a HUD device that uses a light field display (see, for example, Figures 3 to 6).

Patent No. 6580800

The inventor considered light field rendering and as a result recognized the following problems. The following is a step-by-step explanation.

A light field type 3D display device can be constructed, for example, by combining a flat display with an optical component (such as a lenticular lens or a parallax barrier) that has a light beam separation function. The optical component is constructed by arranging basic optical elements (such as microlenses or shielding plates) in the horizontal direction at a certain pitch.

On the other hand, a flat display has a rectangular display area when viewed from above, and the horizontal direction of the rectangle is the horizontal direction, and the vertical direction is the vertical direction. Further, a direction along a line segment (normal line) perpendicular to the display surface of a flat display and a direction away from the display surface is defined as a depth direction.

A plurality of pixels are arranged in a grid pattern in the rectangular display area. One optical element in the above-mentioned optical member corresponds to a group of pixels (or sub-pixels; pixels or sub-pixels may be collectively referred to as "display element unit" or simply "element unit"). Here, explanation will be given in units of pixels.

Among the above group of pixels, 1 to n (n is a natural number greater than or equal to 2, and here n=5 Suppose that there are () pixels. Since the positions of the first to nth pixels are shifted in the horizontal direction by a predetermined distance (for example, the distance of one pixel), the representative light rays emitted by each pixel (for example, among the light emitted by each pixel, Assuming that each light ray passes through the center of the corresponding microlens, each light ray travels while spreading out with directionality while changing its horizontal position.

In the case of a HUD device, each light beam passes through an optical system and is projected onto the vehicle's projected member (windshield, combiner, etc.), and a portion of the light is reflected by the projected member and is transmitted to the viewer (driver, etc.). By entering the eye of the viewer and focusing the image in front of the viewer, a virtual image having a predetermined virtual image display distance is visually recognized.

Here, it is assumed that the virtual image is visible when the viewer's eyes are in the eye box. In addition, the eye box is divided into m pieces at equal intervals in a direction (for example, the width direction of the vehicle) corresponding to the horizontal direction (for example, the arrangement direction of the cylindrical lenses of the lenticular lens) (m is a natural number of 2 or more). :Here, assume a configuration with divided regions of m=5).

Assuming that each light ray emitted from each of the first to nth (n=5) pixels travels ideally without being disturbed, it passes through the optical system and the projected member to the first to nth (n=5) pixels. Each eyebox segmentation region of m is reached.

In this way, ideally, each of the 1st to nth pixels (pixels) at different positions in the horizontal direction and each of the 1st to mth eyeboxes arranged in a direction corresponding to the horizontal direction. A one-to-one correspondence is possible with the divided areas.

However, in reality, the optical system of the HUD device and the projected parts of the vehicle are involved, and if no measures are taken, the path of each light beam will be significantly disrupted.

Here, the position of the curved mirror (concave mirror, etc.) in the optical system may be changed (for example, the concave mirror, etc. may be rotated by an actuator) according to the viewer's eye level, or the large curved mirror may be left unchanged. Furthermore, if the reflective surface area to be used is changed, the reflection position of each light ray on the curved mirror will differ depending on the eye level position. If the reflecting surface is flat, the above relationship will not be significantly broken even if the reflecting surface is moved due to rotation, etc.; however, in reality, concave mirrors etc. ) has a reflective surface in the shape of

The angle of incidence and angle of reflection of the light rays arriving at each reflection position of the curved mirror are different for each light ray, and the curved shape of the curved mirror at each reflection position is also different. Therefore, when a concave mirror, etc. is moved (rotated) according to the eye level position, or when the display position on the display surface of a flat display is changed, the reflection position of the display light on the concave mirror, etc. is changed. , the tendency of the disturbance in the progress of the light rays changes each time.

Since the distance between the left and right eyes of a person is narrow, if the traveling directions of light rays deviate, for example, light that should be incident on the left eye may be incident on the right eye, causing crosstalk. Further, when the light for the left eye and the light for the right eye are reversed, reverse vision occurs.

For example, assume that the eye height positions are classified into three categories: "high", "standard", and "low". Normally, "standard" is considered to be used most frequently, so even if crosstalk etc. is suppressed to some extent by designing based on "standard", it is considered "high" or In the case of "low", it is possible that crosstalk increases.

For example, even if the viewer's eyes remain within the divided area of the eyebox corresponding to "standard", depending on the person's height and posture, The eye height position varies.

For example, the eye level position can be further divided into "high part", "middle part", "bottom part", etc. within the divided area of the eye box. According to each division, the reflection position of the light ray on the curved mirror etc. differs, and the tendency of change in the traveling direction of each light ray differs. Therefore, even within one divided area of an eye box, the degree of crosstalk etc. changes depending on where the eyes are located in the height direction of the vehicle, and it is preferable to take measures against this as well. The above-mentioned problems have been clarified through studies conducted by the present inventor.

One of the objects of the present invention is to suppress the occurrence of crosstalk or the like depending on the eye level position of a viewer in a HUD device.

Other objects of the invention will become apparent to those skilled in the art upon reference to the following illustrative aspects and best embodiment, and the accompanying drawings.

Below, embodiments according to the present invention will be illustrated in order to easily understand the outline of the present invention.
In a first aspect, the head-up display device is a head-up display (HUD) device that displays a virtual image,
a light field type stereoscopic display section that includes a display and an optical member and reproduces display light of a stereoscopic image;
an optical system including a curved mirror that projects the display light reproduced by the three-dimensional display unit onto a projection target member;
a control unit that controls display of images on the display;
When pixels or sub-pixels of the display are used as element units, a storage unit that stores an LF correction value (Light Field rendering calibration parameter) that determines an element unit in which display based on predetermined display data should be performed;
has
The control unit includes:
Light field rendering is performed using the LF correction value determined by at least one of information about the viewer's eye position in the height direction of the vehicle and information from which the eye position can be estimated.

According to the first aspect, even if the correspondence between the first to nth element units (pixels or subpixels) arranged at a predetermined interval in the horizontal direction on the display surface of the display and the divided regions of the first to mth eye boxes arranged in the corresponding horizontal direction is lost, crosstalk can be suppressed by selecting appropriate LF correction value data based on information on the viewer's eye position and at least one piece of information that can estimate the eye position, and performing light field rendering using the LF correction value data.

When the viewer's eye position in the vehicle height direction is changed, for example, the curved mirror may rotate and change the reflection position of the display light, or, even if the large curved mirror does not move, for example, the display position of the image on the display may change, changing the reflection position of the display light on the curved mirror and the path of the light rays, causing the initial correspondence between the pixels and the divided areas of the eye box to be lost. However, it is possible to grasp, for example, at the design stage, how this tends to break down. Therefore, appropriate LF correction can be performed by grasping or estimating the state of the curved mirror (an important optical component) based on information that can estimate the eye position or information on the eye position, and selecting an LF correction value that corresponds to that state.

Here, we will provide a supplementary explanation on the above-mentioned "LF correction value that determines the element unit (pixel or subpixel) to be displayed by the specified display data". For example, the position of the pixel or subpixel for displaying the specified image is determined based on information such as the pitch, thickness, and starting point of the lens of the lenticular lens (however, this is the case when the lenticular lens is obtained as designed). However, in reality, there is manufacturing variation, and the position of the pixel or subpixel is shifted. The calibration function provided in the HUD device as a function to correct this shift is the original LF correction function. However, the shift of the pixel or subpixel position also changes depending on the rotation state of the curved mirror accompanying the change of the eye position, the curvature state of the reflection area used in the curved mirror, etc. In addition, the shift of the pixel or subpixel position is also affected by the warping mode that gives in advance a distortion with an inverse characteristic that cancels the distortion generated by the windshield, etc. Based on such novel knowledge, in the first mode, adaptive LF correction is performed based on the eye position and information that can estimate the eye position (and more preferably, the pre-distortion data is also changed appropriately, but this point will be described later).

In a second aspect subordinate to the first aspect,
The storage unit is configured to store a plurality of images corresponding to eye positions of the viewer that distort an image displayed on the display so as to suppress distortion of a virtual image caused by at least one of the projection target member and the optical system. It stores the pre-distortion data (warping parameters) of
The control unit may switch the LF correction value and the pre-distortion data based on at least one of information on the viewer's eye position in the height direction of the vehicle and information on which the eye position can be estimated. good.

According to the second aspect, by changing the value of the pre-distortion data (warping parameter) in addition to the LF correction value, crosstalk caused by image distortion in the projected member (windshield, etc.) It is also possible to suppress this.

In a third aspect dependent on the first or second aspect,
The information capable of estimating the eye position is information about a setting value of an actuator that drives at least one optical member included in the optical system, or information about a display position of an image on the display of the stereoscopic display unit. ,
The control unit may control the LF correction value corresponding to information on a setting value of the actuator, or the setting information on a display position of an image on the display of the stereoscopic display unit, or the LF correction value and the pre-distortion data. Light field rendering may be performed using .

In the third aspect, information on the actuator settings or setting information on the display position of the image is used as information that can be used to estimate the eye position.

Using this information, we can grasp the condition of the curved mirror, etc., the relative positional relationship between the eye box and the eye, etc., and calculate the LF correction value suitable for the current situation, or the LF correction value and pre-distortion data (warping parameters). ), it is possible to effectively suppress crosstalk and the like.

In a fourth aspect subordinate to the first or second aspect,
The control unit is configured to control the LF correction value determined based on the classification of the position of the viewer's eye in the height direction of the vehicle in an eye box indicating a visible range of the virtual image, or the LF correction value and the above-mentioned LF correction value. Light field rendering may be performed using the predistortion data.

According to the fourth aspect, the data used for rendering can be selected in more detail, for example, by also considering the position of the eye in the height direction (vertical direction) within the divided area of one eyebox. Therefore, it is also possible to reduce (suppress) relatively small crosstalk that may occur depending on the position of the eye in the eye box in the height direction, for example.

In addition, by appropriately setting the number of height position divisions (e.g., high, middle, bottom, etc.) in the eyebox, it is possible to It is possible to reduce the load on the device by adjusting the number of warping parameters (warping parameters) so that they do not increase too much.

Those skilled in the art will readily appreciate that the exemplified embodiments of the present invention may be further modified without departing from the spirit of the present invention.

FIG. 2 is a diagram for explaining the principle of stereoscopic display using a light field method (light beam reproduction method). FIG. 2(A) is a diagram showing an example of the basic configuration of a light field method (light beam reproduction method) stereoscopic display unit (stereoscopic display device) and an example of displaying (generating) a stereoscopic image, and FIG. 2(B) is a diagram showing a stereoscopic display. FIG. 3 is a diagram illustrating an example in which a virtual image is displayed using left-eye and right-eye light rays emitted from a single light point constituting a stereoscopic image in a HUD device using a stereoscopic display device. Figure 3(A) shows a plurality of pixels (5) that correspond to one cylindrical lens and that are arranged at horizontal positions shifted by 1 pixel, and that are arranged at equal intervals in the direction corresponding to the horizontal direction. FIG. 3B is a diagram illustrating the ideal correspondence between the divided areas of a plurality of (5) divided eye boxes, and FIG. 3C are diagrams showing an example of the configuration of pixels (and sub-pixels) in a flat display (an example employing a mosaic arrangement). FIG. 4(A) is a diagram showing the correspondence between pixels and the divided areas of the eyebox in an ideal state (the contents are the same as FIG. 3(A)), and FIG. FIG. 3 is a diagram illustrating an example of the correspondence between each pixel and the divided area of each eyebox after the path of the light ray is disturbed by. FIG. 5 is a diagram illustrating still another example of the correspondence between pixels and divided areas of the eyebox after the path of the light ray is disturbed by an optical system or the like. FIG. 6 is a diagram showing an example of the configuration of a main part of a HUD device that rotates a curved mirror to match the eye height position. FIG. 7 is a diagram illustrating an example of a main part configuration of a HUD device that changes the display position of an image on a display according to the eye level position. FIG. 8(A) is a diagram showing an example of the position of the viewer's eyes in two eyeboxes with different positions in the height direction of the vehicle, and FIG. 8(B) is a diagram showing an example of the position of the viewer's eyes in one vertically long eyebox. FIG. 3 is a diagram showing an example of the position of the viewer's eyes. FIG. 9 is a diagram illustrating an example of setting an LF correction value (or a set of an LF correction value and pre-distortion data) according to the eye height position (division of the eye height position) in the eyebox. FIG. 10 is a diagram illustrating an example of setting the usage area of the LF correction value on a flat display. 11A and 11B are diagrams for explaining pre-distortion correction (warping), and FIG. 11C is a diagram for explaining viewpoint position tracking warping. FIG. 12 is a diagram showing an example of the overall configuration of a virtual image display system equipped with a HUD device (having LF correction and warping functions). FIG. 13 is a diagram illustrating an example of the operation procedure of the HUD device.

The best embodiment described below is used to facilitate understanding of the present invention. Therefore, those skilled in the art should note that the invention is not unduly limited by the embodiments described below.

FIG. 1 is a diagram for explaining the principle of stereoscopic display using the light field method (light beam reproduction method). In FIG. 1, two stereoscopic images IM10 and IM20 are displayed at different depth positions (depth distances) by a light field type stereoscopic display unit (also referred to as a stereoscopic display device.Hereinafter, simply referred to as a stereoscopic display unit) 230. There is.

The stereoscopic display unit (sometimes referred to as a light field display) 230 uses a combination of a flat display 210 using a liquid crystal panel or the like and a lenticular lens 220 as an optical member 220, and uses "lit pixels" ( By controlling the position of the lit pixel (lighted pixel), a defined light beam is reproduced at an arbitrary position and direction.

In addition, the depth position (or depth distance L10, L20) of the stereoscopic images IM10, IM20 when viewed from the stereoscopic display unit 230 side can be adjusted as appropriate by changing the "lighting pattern of lighting pixels (lighting pixels)" on the flat display. , can be adjusted. Note that in FIG. 1, depth distances L10 and L20 are distances from the lens center of the lenticular lens constituting the stereoscopic display section 230 to the respective stereoscopic images IM10 and IM20. However, using the lens center as a reference is just one example, and the depth distance may be defined using another location as a reference.

In addition, in FIG. 1, it is assumed that the left eye AL and the right eye AR are located at the position where the light beams emitted by the three-dimensional images IM10 and IM20 (reproduced light of the three-dimensional images IM10 and IM20) are formed, and from each eye AL and AR. The distances to the respective stereoscopic images IM10 and IM20 are defined as image display distances L30 and L40.

Further, the direction along the line segment connecting the left and right eyes AL and AR of the observer who observes the stereoscopic images IM10 and IM20 in real space (or the viewer who visually perceives the stereoscopic images IM10 and IM20) is defined as the horizontal direction. In FIG. 1, in a flat display 210, a plurality of pixels G are arranged horizontally. Further, in FIG. 1, non-lit pixels are represented by white squares, and lit pixels are represented by black squares.

The lighting (light emission) pattern PT1 of pixels (display pixels) for displaying the stereoscopic image IM10 is a different pattern from the lighting (light emission) pattern PT2 of the display pixels of the stereoscopic image IM20. Due to this pattern difference, the depth distances L10 and L20 of the three-dimensional images IM10 and IM20 can be set to different values.

Also, lighting pattern PT1 is located above lighting pattern PT2 on the paper, and the horizontal positions of the lit pixels (lit pixels) are different. In other words, by adjusting the positions of the lit pixels, the horizontal positions of the two stereoscopic images IM10 and IM20 in three-dimensional space can be made to differ.

See FIG. 2. FIG. 2(A) is a diagram showing an example of the basic configuration of a light field method (light beam reproduction method) stereoscopic display unit (stereoscopic display device) and an example of displaying (generating) a stereoscopic image, and FIG. 2(B) is a diagram showing a stereoscopic display. FIG. 3 is a diagram illustrating an example in which a virtual image is displayed using left-eye and right-eye light rays emitted from a single light point constituting a stereoscopic image in a HUD device using a stereoscopic display device. In FIG. 2, parts common to those in FIG. 1 are given the same reference numerals (this point also applies to the following figures).

The flat display 210 has a rectangular area (display area, or display surface) 215 in a plan view. A large number of pixels (picture elements) are arranged two-dimensionally in a grid pattern on this display surface 215. The horizontal direction is the width of the rectangular display surface 215. The vertical direction is the length of the display surface 215. The depth direction is the direction along a line segment (normal line) perpendicular to the display surface 215 and away from the display surface 215.

In addition, in FIG. 2A, on the display surface 215, there are pixels IML used to generate display light EL(1) for the left eye, and pixels IML used to generate display light ER(1) for the right eye. The resulting pixel IMR is shown.

As the optical member RS, in addition to the lenticular lens 220 shown above, a parallax barrier (parallax barrier) 225 may be used. However, these are examples and are not limited to these.

As shown on the upper left side of FIG. 2(A), the lenticular lens 220 has a configuration in which cylindrical lenses are arranged horizontally at a predetermined pitch, and separates light rays for each of the left and right eyes. Has a light beam separation function.

Further, the parallax barrier 225 is, for example, thin strip-shaped (rectangular) shielding objects 225a to 225n arranged horizontally at a predetermined pitch while providing gaps (slits) SL. Even if the image is the same, by blocking part of the image display light with a blocking object, different image display light (directional display light) is generated for each eye. be. It is similar to a lenticular lens in that it separates light rays for each eye.

The light separated by the optical member RS having a beam separation function (in other words, the display lights E(L1) and E(R1) for each eye that display a stereoscopic image) is transmitted to both eyes AL located at the light imaging point. , when light enters the AR, a person sees an apparent three-dimensional image IM at a location where convergence (intersection of light) occurs. In other words, this can be considered to mean that the stereoscopic image IM is displayed (or generated) by the stereoscopic display unit 230. Note that in the example of FIG. 2(A), the convergence angle is θc.

Next, refer to FIG. 2(B). In FIG. 2(B), an eyebox EB is set in front of a viewer (such as a vehicle driver) who is a user of the HUD device 100. Note that in the figure, the eyebox EB is simplified and drawn in a two-dimensional manner.

Eyepoint P(C) is located at the center of eyebox EB. Assuming that virtual imaging planes PS (L) and PS (R) corresponding to the left and right eyes are set in front of the windshield 2, the virtual image V (C) is located in the center of the overlapping area. . The convergence angle of the virtual image V(C) is θd, and the virtual image V(C) is recognized by the viewer (user) as a three-dimensional image.

This three-dimensional virtual image V(C) is displayed (formed) as follows. That is, the optical system of the HUD device includes display lights E(L1) and E(R1) for each of the left and right eyes of the virtual stereoscopic image IM generated by the 3D display shown in FIG. 2(A). The display lights E (L1) and E (R1) are reflected by a curved mirror (concave mirror, etc.) 171 (the number of reflections is at least once) and projected (projected) onto the windshield 2, and the reflected light is reflected by the viewer. The virtual image V(C) is displayed (formed) by focusing the image in front of the windshield (projection target member) 2.

Next, refer to FIG. Figure 3(A) shows a plurality of pixels (5) that correspond to one cylindrical lens and that are arranged at horizontal positions shifted by 1 pixel, and that are arranged at equal intervals in the direction corresponding to the horizontal direction. FIG. 3B is a diagram illustrating the ideal correspondence between the divided areas of a plurality of (5) divided eye boxes, and FIG. 3C are diagrams showing an example of the configuration of pixels (and sub-pixels) in a flat display (an example employing a mosaic arrangement). Note that parts common to FIGS. 1 and 2 are denoted by the same reference numerals (this point also applies to subsequent drawings).

In FIG. 3A, five pixels (element units) G10 to G14 are arranged on the display surface 215 of the flat display 210, shifted by one pixel in the horizontal direction, for one cylindrical lens of the lenticular lens 220, and these are the pixels to be lit. Furthermore, the pixels G10 to G14 are lit up, for example, based on display data α (for example, display data for the right-eye image).

In the example of FIG. 3(A), the pixels G10 to G14 are arranged diagonally instead of in a horizontal row, but this is because when these pixels are turned on, the pixel density in the horizontal and vertical directions is not balanced. This takes into consideration that the light rays can be properly dispersed.

As shown in FIG. 3(A), the light emitted from each of the pixels G10 to G14 travels in the depth direction while changing the position in the horizontal direction, and as explained earlier, the light emitted from each of the pixels G10 to G14 travels in the depth direction while changing the position in the horizontal direction. (However, it is omitted in FIGS. 3A and 3B) to reach the eyebox EB.

The eye box EB is located in a direction corresponding to the horizontal direction of the stereoscopic display section 230 in the real space where the viewer (driver, etc.) is located (the width direction of the vehicle, or the left and right direction along the line connecting the left and right eyes). It is divided into five areas along the Each area is referred to as a divided area of the eyebox, and is indicated by reference numerals J10 to J14 in FIG. 3(A). Note that the eyebox EB is drawn in a two-dimensional manner for the sake of simplification. Furthermore, the viewer's eye (here, the right eye) A is located in the divided area J11 of the eye box.

In an ideal situation where the light rays are not disturbed at all, each of the pixels G10-G14 would reach each of the eyebox divisions J10-J14. In other words, each pixel G10 to G14 and each eyebox divided region J10 to J14 have a 1:1 correspondence relationship.

As shown in FIG. 3B, the HUD device 100 including the stereoscopic display section 230 can be viewed as a multi-view virtual image display device with five viewpoints.

FIG. 3C shows an example of the configuration of pixels (and subpixels) in a flat display (an example employing a mosaic arrangement). Five pixels are arranged in the horizontal direction for one cylindrical lens of the lenticular lens 220. One pixel is composed of sub-pixels of three colors: R (red), G (green), and B (blue).

Pixel G10 is composed of RGB sub-pixels, pixel G11 is composed of BRG sub-pixels, pixel G12 is composed of GBR sub-pixels, pixel G13 is composed of RGB sub-pixels, and pixel G14 is composed of RGB sub-pixels. It is composed of BRG subpixels.

Next, refer to FIG. 4. FIG. 4(A) is a diagram showing the correspondence between pixels and the divided areas of the eyebox in an ideal state (the contents are the same as FIG. 3(A)), and FIG. Figure 4(B) shows an example of the correspondence between each pixel and the divided area of each eyebox after the path of the ray is disturbed by the optical system, etc. , FIG. 7 is a diagram showing another example of the correspondence between each pixel and each divided area of each eyebox.

In the ideal state of FIG. 4A, as described above, there is a 1:1 correspondence between each pixel G10 to G14 and each eyebox divided region J10 to J14.

In the state shown in FIG. 4B after the path of the light ray is disturbed, the correspondence between the pixels and the divided areas of the eyebox is broken, and here, each of the pixels G8 to G12 corresponds to the divided area J10 of the eyebox. -J14.

Here, if we continue to light up pixels G10 to G14 as shown in FIG. 4(A) without considering the changes shown in FIG. 4(B), the reproduced light of the image for the right eye based on the data α is is not incident, and the light from pixel G9 reaches right eye A. For example, if the pixel G9 emits the light of the left-eye image based on different display data, the light of the left-eye image will be incident on the right eye A, and crosstalk will occur.

However, in FIG. 4(B), the pixels that display the right eye image based on the data α are shifted from G10 to G14 in FIG. 4(A) to G8 to G12 in consideration of the change in the correspondence relationship. There is.

In other words, the lit (illuminated) pixel based on data α is shifted by two pixels in the horizontal direction. The correction amount for the lit pixel is two pixels in the horizontal direction. This can be seen as the result of a correction that sets the LF correction value (written as LFC in the diagram) to "+2" and moves the position of the lit pixel (or the position of the lighting pattern) by +2 pixels in the horizontal direction.

After this LF correction is performed, pixel G9 displays the right eye image, so light from the right eye image enters the right eye A, similar to FIG. 4(A), and crosstalk occurs. do not have. In this way, taking into account the disturbance (collapse) in the correspondence between pixels and the divided areas of the eyebox, the position of the lit pixel (or Crosstalk can be suppressed by performing a correction that shifts the position of the lighting pattern in the horizontal direction.

Note that in FIG. 4B, the LF correction value is shown as the amount of deviation from the lit pixel when no correction is made, but this is just an example. For example, pixels G8 to G12 to be lit may be determined by specifying "pixels or subpixels for one pitch from G8". Various ways of specifying a pixel or subpixel (element unit) are conceivable. The designation method may change depending on the situation. Moreover, a plurality of values can also be used as the LF correction value.

Next, refer to FIG. 5. FIG. 5 is a diagram illustrating still another example of the correspondence between pixels and divided areas of the eyebox after the path of the light ray is disturbed by an optical system or the like.

In FIG. 5, as a result of the path of the light rays being disturbed, a situation has arisen in which pixels G15-G19 correspond to divided regions J10-J14 of the eyebox. If the display shown in FIG. 4A is continued without considering this situation, crosstalk will occur.

Therefore, in the example of FIG. 5, the LF correction value (LFC) is set to "-5", and the pixels G15 to G19 to be displayed based on the data α are determined (designated). In other words, the lighting pixels based on the data α are changed (corrected) from G10 to G14 in FIG. 4A to G15 to G19 in FIG. This makes it possible to maintain an appropriate correspondence between pixels and sub-pixels and the divided areas of the eyebox, thereby suppressing the occurrence of crosstalk.

Next, refer to FIG. FIG. 6 is a diagram illustrating an example of a main part configuration of a HUD device that rotates a curved mirror according to the eye level position.

The HUD device 100 includes a three-dimensional display section 230, a folding mirror (reflector) 133, a curved mirror (here, a concave mirror) 131, an I/O interface for acquiring information from external sensors and other ECUs, and a processor. , a memory, and a control unit (HMI control unit) 300 configured from a computer program stored in the memory.

The angle of the concave mirror 131 can be adjusted as appropriate by the operation of a rotation mechanism 129 consisting of an actuator. The actuator 129 is driven by an actuator driver 189.

The control unit 300 controls the operation of the stereoscopic display unit 230, the actuator drive unit 189, and the like. The control unit 300 sets, for example, an actuator setting value SD that determines the amount of rotation (rotation) of the curved mirror (concave mirror) 131 in the actuator drive unit 189.

In the example of FIG. 6, the height position of the eyes (view point) A of the viewer (driver, etc.) varies depending on the viewer's height, driving posture, and the like. As an example, assume that there are three eye (viewpoint) height positions: middle (center), up (high), and down (low). Furthermore, the height position of the eye box EB is also changed depending on the eye height.

In the example of FIG. 6, the curved mirror (concave mirror) 131 is rotated according to the height (UP, MD, DW) of the eye (viewpoint) A, and the course of the display light K is changed. As a result, no matter where the eye (viewpoint) A is located, the display light K reflected by the windshield 2 will be incident on the eye (viewpoint) A.

However, as described above, when the curved mirror (concave mirror, etc.) 131 is rotated, the reflection position of the light beam changes, and the direction of reflection is disturbed, making crosstalk likely to occur. For this reason, LF correction is performed, but in order to perform appropriate LF correction, it is necessary to select an LF correction value depending on the current state of the optical system.

The current state of the optical system can be understood by knowing the state of the curved mirror 131, which is involved in the generation of crosstalk. Since the curved mirror 131 rotates according to the eye height position, information on the eye height position is useful information for knowing the state of the curved mirror 131.

In addition, the actuator setting value SD set in the actuator drive unit 189 for rotating the curved mirror 131 is information that allows estimating the eye height position, and is also used to accurately know the state of the curved mirror 131. It is also a powerful piece of information.

In the example of FIG. 6, the actuator setting value SD that the control unit 300 sets in the actuator driving unit 189 indicates the amount of rotation of the curved mirror 131, and this amount of rotation corresponds to the position of the eye (viewpoint) A. As a result, the actuator setting value SD can be referenced as information that can be used to estimate the eye position, and the eye position (eye height position) can be estimated. Therefore, it is possible to identify or estimate the eye position using at least one of the actually measured eye position and information that can be used to estimate the eye position, and use the result for light field correction.

Next, refer to FIG. FIG. 7 is a diagram illustrating an example of a main part configuration of a HUD device that changes the display position of an image on a display according to the eye level position.

In FIG. 7, parts that are common to those in FIG. 6 are given the same reference numerals, but a dash mark is given in FIG. 7 so that they can be distinguished. Although the basic configuration is the same as that in FIG. 6, in FIG. No means are provided for rotating 131'. Therefore, there is no need to take into account errors caused by the rotation of the curved mirror 131'.

In the example of Figure 7, a change in the height position of eye (viewpoint) A is accommodated by changing the display position of the image on the display surface 215' of the flat display (display unit) 210' included in the stereoscopic display unit 230'.

In the example of FIG. 7, the display position of the image changes as points P1'', P1, and P1' in the figure, depending on the height (UP, MD, DW) of the eye (viewpoint) A. The amount of change is D1. Therefore, for example, the amount and direction of deviation of the image position (in other words, the setting information of the display position of the image on the display) based on the position P1 when the height of the eye (viewpoint) A is at middle (MD). By detecting , the height position of the eye (viewpoint) A can be estimated.

Therefore, the amount of change D1 (in other words, the setting information of the display position of the image on the display) can be used as information from which the height position of the eye (viewpoint) A can be estimated. For example, the position of the eye in the eye box in the height direction of the vehicle can be estimated using the information on the amount of change in position D1.

Next, refer to FIG. FIG. 8(A) is a diagram showing an example of the position of the viewer's eyes in two eyeboxes with different positions in the height direction of the vehicle, and FIG. 8(B) is a diagram showing an example of the position of the viewer's eyes in one vertically long eyebox. FIG. 3 is a diagram showing an example of the position of the viewer's eyes.

In FIG. 8(A), eyebox EB20 is located above eyebox EB10. Eyebox EB10, located below, has eyebox divided regions J1 to J5. Eyebox EB20, located above, has eyebox divided regions J21 to J25.

Eye (viewpoint) A is located above the divided area J4 of EB10 and located below the divided area J24 of EB20. Even though the heightwise position of the eye (viewpoint) A is the same, the heightwise position of the eyeboxes EB10 and EB20 is different, resulting in a relative position of the eye (viewpoint) with respect to each eyebox EB10 and EB20. will be different.

Taking this into consideration, for example, the data used for rendering can be refined in more detail by also taking into account the position of the eyes (classification of eye positions) in the height direction (up and down direction) within the divided area of one eyebox. You can choose. This makes it possible to reduce (suppress) relatively small crosstalk that may occur depending on the position of the eye in the eye box in the height direction.

In addition, by appropriately setting the number of height divisions (e.g., high, middle, bottom, etc.) within the eyebox, it is possible to adjust the number of LF correction values and pre-distortion data (warping parameters) prepared in advance so that they do not increase too much, thereby reducing the burden on the device.

In FIG. 8(B), the eyebox EB30 (having divided areas J31 to J35, and the eye is located within J34) does not move. However, within the eye box EB30, the eye height position changes from A-1 to A-2 to A-3. In this case as well, the relative positional relationship between the eyebox and the eye is different. Taking this into consideration, for example, the data used for rendering can be refined in more detail by also taking into account the position of the eyes (classification of eye positions) in the height direction (up and down direction) within the divided area of one eyebox. can be selected, and the same effect as in the case of FIG. 8(A) can be obtained.

Next, reference is made to FIG. 9. FIG. 9 is a diagram showing an example of setting the LF correction value (or a set of the LF correction value and pre-distortion data) according to the eye height position (classification of the eye height position) within the eyebox.

In FIG. 9, it is assumed that the position of the eyebox changes as in the example of FIG. 8(A). The positions (states) of the curved mirror (in this case a concave mirror) are "Shorter" corresponding to a viewpoint position (viewpoint height position) lower than the standard, "Nominal" corresponding to the standard viewpoint position, and "Nominal" corresponding to the standard viewpoint position, and "Nominal" corresponding to the standard viewpoint position. It is divided into "Torah" corresponding to. Further, the height position of the viewpoint within the eyebox (in other words, the height position of the viewpoint relative to the eyebox) is classified (classified) into high, middle, and bottom.

A high part, a middle part, and a bottom part are set for each of the shorter, nominal, and toller. As a result, a total of nine types (minimum nine) of LF correction values (or a set of LF correction values and pre-distortion data) of A1 to A3, B1 to B3, and C1 to C3 are stored in the storage section (as shown in FIG. 12). 218). Note that it is possible to use different LF correction values for each of the left and right eyes.

FIG. 10 is a diagram illustrating an example of setting the usage area of the LF correction value on a flat display. In the example of FIG. 10, a usage area 400 is set as an area where the LF correction value is used (in other words, a range of pixels and sub-pixels that are targets of LF correction using the LF correction value). By appropriately setting the usage area, the burden of LF correction processing can be reduced. This is advantageous, for example, in performing high-speed arithmetic processing.

Next, let us refer to FIG. 11. FIGS. 11(A) and (B) are diagrams for explaining pre-distortion correction (warping), and FIG. 11(C) is a diagram for explaining viewpoint-tracking warping.

As shown in FIG. 11A, even when the virtual image V is displayed on the flat virtual image display surface PS, the virtual image is The surface deforms like PS'. Similarly, virtual image V deforms like virtual image V'. Warping (pre-distortion correction) is performed to reduce this deformation as much as possible.

Warping (pre-distortion correction) is a pre-distortion method that applies distortion to the image data (light field image data) in advance, which has the opposite characteristics to the distortion caused by the above-mentioned optical system, windshield (projection target member) 2, etc. (pre-distortion method) correction.

As shown in FIG. 11(B), a plurality of reference quasi-coordinates GD (i, j) are set in the image area to be warped, and by performing warping processing on each reference coordinate point, The entire image area can be distorted so as to have characteristics opposite to those actually caused by an optical system or the like.

The viewpoint position tracking warping process will be explained using FIG. 11(C). As shown in FIG. 11(C), the eye box EB is divided into multiple (here, nine) divided regions J1'-J9', and the position of the viewpoint A of the viewer (driver) for warping is detected for each divided region J1'-J9'.

Display light K is emitted from the optical system of the HUD device 100, a portion of which is reflected by the windshield 2, and enters the eyes (view point) A of the viewer (driver). When the eyes (view point) A are within the eye box EB, the viewer (driver) can visually recognize the virtual image of the image.

The HUD device 100 has a ROM 215, and the ROM 215 includes an image conversion table 217. The image conversion table 217 stores, for example, warping parameters WP that determine polynomials, multipliers, constants, etc. for image correction (warping image correction) using a digital filter. The warping parameter WP is provided corresponding to each of the partial areas J1 to J9 in the eyebox EB. In FIG. 11C, warping parameters corresponding to each partial area are shown as WP(J1') to WP(J9'). In the figure, only WP (J1'), WP (J4'), and WP (J7') are shown as symbols.

When the eye (viewpoint) A moves, the position of the viewpoint A among the plurality of partial regions J1' to J9' is detected. Then, one of the warping parameters WP(J1') to WP(J9') corresponding to the detected partial area is read out from the ROM 215 (warping parameter update), and warping processing is performed using the warping parameter. Ru.

Thereby, warping is performed according to the viewpoint position, and image distortion caused by the optical system, the windshield 2, etc. can be effectively reduced (suppressed). Warping is a process that also helps prevent crosstalk.

Therefore, in a preferred embodiment of the present invention, the above-described LF correction value and warping parameter are switched in the same frame (in the same frame period). In other words, the LF correction value and the warping parameter are set as a set and switched within a frame, and the warping is also used for reinforcing the LF correction. Thereby, it is possible to further reduce the mismatch between the above-mentioned pixels (or sub-pixels) and the divided areas of the eyebox.

As described above, in some embodiments, as shown in FIG. J7' to J9'), the warping parameters also change. That is, when the eye (viewpoint) A changes in the width direction of the vehicle, the light field rendering unit 193, which will be described later, can change only the warping parameter WP without changing the LF correction value.

Next, refer to FIG. 12. FIG. 12 is a diagram showing an example of the overall configuration of a virtual image display system equipped with a HUD device (having a function of performing position correction and warping). In FIG. 12, parts common to those in the previous drawings are given the same reference numerals. Description of the configuration described above will be omitted.

In the example of FIG. 12, the vehicle 1 is provided with a peripheral imaging camera 187 that images the periphery (including the front) of the vehicle 1, and a pupil imaging camera 188.

The control unit (HMI control unit) 300 includes a viewpoint position detection unit 192, a light field rendering unit 193, a ROM 215, a VRAM 201, and an image drawing unit 202.

The light field rendering unit 193 includes an LF correction unit 194 that performs LF correction using the LF correction value described above, and a pre-distortion correction unit (warping processing unit) 195 that performs pre-distortion correction (warping). has.

Further, the ROM 215 stores a warping parameter table (image conversion table) 217 and an LF correction value table 218. The warping parameter table (image conversion table) 217 and the LF correction value table 218 have, for example, a data structure in the format shown in FIG. 9 above.

Original image data (eg, samples of multi-view images) 196 can be stored in the VRAM 201 . Furthermore, data after rendering may also be temporarily stored in the VRAM 201 .

The image drawing unit 202 draws light field data on the display unit 210 included in the stereoscopic display unit 230. In other words, light field drawing processing is performed.

In the example of FIG. 12, an image processing unit 302 is provided to process the image captured by the peripheral image capturing camera 187 to determine objects, etc. Also provided is a vehicle ECU 310 that collects various vehicle information, etc. Also provided is a communication unit 320 that has a GPS communication function and various communication functions, and a navigation ECU 330 that can provide various navigation information.

Further, various types of information can be provided to the light field rendering unit 193 of the control unit 300 from each of the image processing unit 302, vehicle ECU 310, and navigation ECU 330.

In the example of FIG. 12, the display light E1 displays the first virtual image (virtual image of the image set up against the road surface 40) V1, and the display lights E2 and E3 display the second virtual image (the virtual image of the road surface 40). A virtual image (virtual image of the image superimposed on the image) V2 is displayed.

Next, refer to FIG. 13. FIG. 13 is a diagram illustrating an example of the operation procedure of the HUD device. In step S1, information indicating the viewpoint position (including information that allows estimation of the viewpoint position) is input to the control unit 300.

In step S2, for example, the position of the concave mirror (curved mirror) 131 is determined according to the viewpoint position. In step S3, LF correction value data corresponding to the position of the concave mirror (curved mirror) 131 is selected.

In step S4, the usage area of the LF correction data is determined (specified) (see FIG. 10). In step S5, pre-distortion data (warping parameters) are selected depending on the position of the concave mirror (curved mirror), the viewpoint position, and the like.

In step S6, light field data is generated by rendering. At this time, LF correction processing and warping processing (warping correction) are also performed. In other words, light field rendering with LF correction and warping is performed, thereby generating light field data that is display data of an image (for example, parallax images for the left and right eyes).

In step S7, light field data is drawn on the display section 210 of the stereoscopic display section 230. As a result, pixels or sub-pixels at predetermined positions on the display surface 215 of the display 210 emit light in a predetermined pattern (see FIG. 1), thereby displaying a three-dimensional image (see FIGS. 1 and 2(A)). ), the display light enters the human eye via an optical system, a windshield, etc.

As described above, according to the present invention, it is possible to suppress the occurrence of crosstalk or the like depending on the eye level position of the viewer in the HUD device.

In this specification, the term vehicle can also be broadly interpreted as a vehicle. In addition, terms related to navigation (for example, signs, etc.) shall be interpreted in a broad sense, taking into account, for example, the perspective of navigation information in a broad sense that is useful for vehicle operation. Furthermore, HUD devices include those used as simulators (for example, aircraft simulators, game consoles, etc.). Further, the types of the display section and optical members are not limited to those in the above embodiments, and various types can be adopted.

The invention is not limited to the exemplary embodiments described above, and those skilled in the art will be able to easily modify the exemplary embodiments described above to the extent that they fall within the scope of the claims. .

DESCRIPTION OF SYMBOLS 1... Vehicle (own vehicle), 2... Projected member (windshield etc.), 100... HUD device (3D display (3D) HUD device), 129... Actuator, 131... Curved surface Mirror (concave mirror, etc.), 188... Peripheral imaging camera, 188... Pupil imaging camera (or face imaging camera), 189... Actuator drive section, 192... Viewpoint position detection section, 193... Light Field rendering section, 194... LF correction section, 195... Pre-distortion correction section (warping processing section), 217... Warping parameter (pre-distortion correction) table, 218... LF correction value table, 196. ...Original image data, 202... Image drawing unit, 210... Display (flat display), 215... Display surface, 220... Optical member as a light beam separation unit (lenticular lens, etc.), 230... ... Stereoscopic display unit (3D display unit), 300... Control unit (HMI control unit), 302... Image processing unit, 310... Vehicle ECU, 320... Communication unit, 330... Navigation ECU

Claims (2)

A head-up display (HUD) device that displays a virtual image,
a light field type stereoscopic display section that includes a display and an optical member and reproduces display light of a stereoscopic image;
an optical system including a curved mirror that projects the display light reproduced by the three-dimensional display unit onto a projection target member;
a control unit that controls display of images on the display;
When pixels or sub-pixels of the display are used as element units, a storage unit that stores an LF correction value (Light Field rendering calibration parameter) that determines an element unit in which display based on predetermined display data should be performed;
has
The control unit includes:
The LF correction value is determined based on the state of the curved mirror and the classification of the position in the eye height direction of the viewer in an eye box indicating a visible range of a virtual image whose position changes depending on the state of the curved mirror. Perform light field rendering using
A head-up display device characterized by: The storage unit is configured to store a plurality of images corresponding to eye positions of the viewer that distort the image displayed on the display so as to suppress distortion of the virtual image caused by at least one of the projection target member and the optical system. It stores the pre-distortion data (warping parameters) of
The control unit switches the LF correction value and the pre -distortion data based on the state of the curved mirror and the division in the eyebox whose position changes depending on the state of the curved mirror .
The head-up display device according to claim 1, characterized in that:

Images