JP2020083280A - Image display system, movable body, image display method, and image display device - Google Patents

Abstract

To perform image display having augmented reality without requiring calibration.SOLUTION: An image display system equipped with an imaging device and an image display device is provided to a movable body. The imaging device includes an imaging unit for capturing an image in which the head of a crew member of the movable body and the outside of the movable body are included in the same image. The image display device includes an image creation unit for creating a display image, an image display unit for displaying the created display image at a position overlapping with the outside of the movable body when visually recognized by the crew member, and a changing unit for changing at least one of a position and an inclination for displaying the display image according to the image.SELECTED DRAWING: Figure 10

Description

The present invention relates to an image display system, a moving body, an image display method, and an image display device.

2. Description of the Related Art A head-up display (hereinafter referred to as HUD), which is provided in a moving body such as a vehicle and projects an image directly on the field of view of an occupant such as a driver, has been known.

In the HUD, the projection position of the image of the HUD is corrected according to the vibration information of the vehicle detected by the acceleration sensor and the posture information of the occupant acquired by the image capturing unit to obtain an augmented reality (AR) feeling. A technique for displaying a certain image is disclosed (for example, refer to Patent Document 1).

However, in the technique of Patent Document 1, the projection position of the image of the HUD is corrected based on the information output by the different components of the acceleration sensor and the image pickup unit. It was necessary to calibrate, which was time-consuming.

The present invention has been made in view of the above, and an object of the present invention is to display an image with augmented reality without the trouble of calibration.

An image display system according to an aspect of the disclosed technology is an image display system that includes an imaging device and an image display device and is provided in a moving body, wherein the imaging device is a head of an occupant of the moving body. And an outside of the moving body have an image pickup unit that picks up an image included in the same image, and the image display device includes an image generation unit that generates a display image, and the generated display. An image display unit that displays an image at a position overlapping the outside of the moving body when visually recognized by the occupant, and a changing unit that changes at least one of a position and an inclination at which the display image is displayed according to the image. And have.

According to the disclosed technology, it is possible to display an image with an augmented reality without the need for calibration.

It is a figure explaining an example of composition of HUD concerning an embodiment, (a) is a figure explaining an example of arrangement in a vehicle of HUD, and (b) is a pitching angle of a vehicle, a yaw angle, and a roll angle. It is a figure explaining. It is a figure explaining an example of composition of an image display part concerning an embodiment. It is a block diagram explaining an example of composition of an image display system of a vehicle carrying HUD. It is a block diagram explaining an example of hardware constitutions of a treating part concerning an embodiment. It is a figure explaining an example of a display image concerning an embodiment, (a) is a figure explaining an example of a display image of a mark, and (b) is a figure explaining an example of a display image of course information. It is a figure explaining an example of a position gap of a display picture by pitching of vehicles, (a) is a figure explaining pitching of vehicles, (b) is a figure explaining a normal display picture, (c). FIG. 6 is a diagram for explaining a positional shift of a display image due to pitching. It is a figure explaining an example of the inclination gap of the display image by rolling of a vehicle, (a) is a figure explaining rolling of a vehicle, and (b) is a figure explaining the inclination gap of a display image by rolling. It is a figure explaining an example of the position gap of a display image by movement of a driver's head, (a) is a figure explaining an image in which a driver's head is included, and (b) is a movement of a head. FIG. 6C is a diagram for explaining the positional deviation of the display image due to the movement of the head. It is a figure explaining an example of arrangement in a vehicle of a wide-angle camera concerning a 1st embodiment, (a) is a side view of a vehicle, and (b) is a top view of a vehicle. It is a figure explaining an example of the detection image which concerns on 1st Embodiment, (a) is a figure explaining the detection image in which the exterior of a vehicle and a driver|operator's head are contained in the same image, (b) FIG. 3 is an enlarged view illustrating details of the outside of the vehicle. It is a block diagram explaining an example of functional composition of an image display system concerning a 1st embodiment. It is a sequence diagram explaining an example of operation|movement of the image display system which concerns on 1st Embodiment. It is a figure explaining an example of a display image after position gap amendment, (a) is a figure explaining a display image after position gap amendment by pitching of a vehicle, and (b) is after tilt gap amendment by rolling of a vehicle. 6C is a diagram for explaining the display image of FIG. 6C, and FIG. 7C is a diagram for explaining the display image after positional displacement correction due to the movement of the head of the occupant. It is a figure explaining the 1st example of the position gap correction effect of the display image by a slope, (a) is a figure explaining the case of an uphill, and (b) is a figure explaining the case of a downhill. It is a figure explaining the 2nd example of the position gap correction effect of the display image by a slope. It is a figure explaining the 1st example of position gap amendment effect of the display image of course information, (a) is a figure explaining the display image of course information before amendment, (b) is the course information after amendment. It is a figure explaining a display image. It is a figure explaining the 2nd example of the position gap correction effect of the display image of track information, (a) is a figure explaining a slope, (b) is a display image of track information after position shift correction by a slope. It is a figure explaining. It is a figure explaining an example of the detection method by the image display system concerning the modification of a 1st embodiment, (a) is a figure explaining a normal display image, and (b) is a figure of a detected display image. It is a figure explaining the gap of a position and inclination. It is a figure explaining an example of composition of an image display system concerning a 2nd embodiment. It is a block diagram explaining an example of functional composition of an image display system concerning a 2nd embodiment. It is a figure explaining an example of the detection image by the stereo camera which concerns on 2nd Embodiment, (a) is a figure explaining the detection image by a 1st imaging part, (b) is a detection image by a 2nd imaging part. It is a figure explaining. It is a sequence diagram explaining an example of operation of the image display system concerning a 2nd embodiment. It is a figure explaining an example of the display image concerning a 3rd embodiment.

Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In each drawing, the same reference numerals are given to the same components, and duplicate description may be omitted.

The image display system according to the embodiment is a system that includes an imaging device and an image display device and is provided in a moving body such as a vehicle. In the embodiment, an example will be described in which a vehicle, which is an example of a moving body, is provided with a wide-angle camera, which is an example of an image capturing device, and a head-up display (hereinafter, referred to as HUD), which is an example of an image display device.

<HUD configuration>
First, the configuration of the HUD according to the embodiment will be described.

The HUD is an image display device that visually displays navigation information necessary for operating the vehicle to a driver (an example of an occupant) of the vehicle through a windshield of the mounted vehicle. Here, the navigation information is information such as the speed of the vehicle, route information, distance to the destination, current location name, presence/absence or position of an object in front of the vehicle, signs such as speed limits, and traffic jam information. The image including the navigation information displayed by the HUD is an example of “display image”, and will be simply referred to as a display image hereinafter.

FIG. 1 is a diagram illustrating an example of a configuration of a HUD according to an embodiment, FIG. 1A is a diagram illustrating an example of arrangement of a HUD in a vehicle, and FIG. 1B is a pitching angle and a yaw angle of the vehicle. It is a figure explaining a roll angle.

As shown in FIG. 1A, the HUD 100 is mounted on a vehicle 120 and has a processing unit 101 and an image display unit 102.

The processing unit 101 has a function of generating a display image including the above-mentioned navigation information and outputting the display image to the image display unit 102. The processing unit 101 can be arranged at any position in the vehicle 120.

The image display unit 102 has a function of superimposing the display image generated by the processing unit 101 in the vehicle 120 on the visual field of the driver 130 as a virtual image and displaying the superimposed image. In other words, the image display unit 102 can display the input display image at a position overlapping the outside of the vehicle 120 when viewed by the driver 130.

The image display unit 102 may be arranged at any position according to the interior design of the vehicle 120, and can be arranged on the dashboard inside the vehicle 120. The image display unit 102 may be arranged on the ceiling or the like in the vehicle 120. Alternatively, the image display unit 102 may be embedded in the dashboard.

More specifically, the image display unit 102 can display the internally generated intermediate image as a magnified virtual image using a mirror, a lens, or the like, and display the image with a certain sense of distance from the viewpoint of the driver 130.

The image display unit 102 can be implemented by a "panel method" in which an intermediate image is formed by an imaging device such as a liquid crystal panel, a DMD (Digital Micro-mirror Device) panel, or a fluorescent display tube (VFD; Vacuum Fluorescent Display). Although there is a “laser scanning method” in which a laser beam emitted from a laser light source is scanned by a two-dimensional scanning device to form an intermediate image, any of the modes may be used in the embodiment.

However, the laser scanning method is suitable for use in the embodiment because it can widen the angle of the virtual image and can display a high-luminance image that is robust against external light. In the embodiment, the laser scanning image display unit 102 will be described as an example.

FIG. 1B shows a pitching angle, a yaw angle, and a roll angle of the vehicle 9. Rolling (rotating or tilting) an object such as a moving object whose front, rear, left and right, and top and bottom are rotated (or tilted) about the front and rear axes (Z axis in the figure), and rotation about the left and right axes (X axis in the figure) ( Or tilting is called pitching, and rotating (or tilting) about the vertical axis (Y axis in the figure) is called yawing. In addition, each rotation amount or inclination amount is referred to as a roll angle, a pitching angle, or a yaw angle.

Next, FIG. 2 is a diagram illustrating an example of the configuration of the image display unit according to the embodiment. As shown in FIG. 2, the image display unit 102 includes a light source unit 51, a light deflector 52, a first mirror 53, a scanned surface 54, and a second mirror 55. In FIG. 2, reference numeral 135 denotes an eyeball of the driver 130 (hereinafter referred to as an eyeball 135), and 160 denotes a virtual image (hereinafter referred to as a virtual image 160).

The light source unit 51 includes three laser light sources corresponding to RGB, a coupling lens, an aperture, a synthesizing element, a lens, and the like, and synthesizes the laser light emitted from the three laser light sources to reflect the light from the optical deflector 52. Lead towards the surface. The laser light guided to the reflecting surface of the light deflector 52 is two-dimensionally deflected by the light deflector 52. The light source unit 51 is composed of three LDs (Laser Diodes) corresponding to RGB.

As the optical deflector 52, it is possible to use one minute mirror that oscillates with respect to two orthogonal axes, or two minute mirrors that oscillate or rotate about one axis. The optical deflector 52 can be a MEMS (Micro Electro Mechanical Systems) manufactured by a semiconductor process or the like. The optical deflector 52 can be driven by an actuator that uses the deforming force of the piezoelectric element as a driving force.

The light beam deflected two-dimensionally by the optical deflector 52 is incident on the first mirror 53, is folded back by the first mirror 53, and draws a two-dimensional image on the scanned surface 54. A concave mirror or the like can be used as the first mirror 53. Further, the reflecting surface of the first mirror 53 may have an anamorphic shape. That is, the reflecting surface of the first mirror 53 can be a reflecting surface having a shape in which the curvature in the predetermined direction and the curvature in the direction orthogonal thereto are different. By making the reflecting surface of the first mirror 53 an anamorphic shape, the aberration correction performance can be improved.

The scanned surface 54 is a transparent surface on which the light flux reflected by the first mirror 53 is incident to form a two-dimensional image. The surface 54 to be scanned has a function of diverging the combined laser light at a desired divergence angle, and is preferably a microlens array structure or the like. The light flux emitted from the surface 54 to be scanned is enlarged and displayed by the second mirror 55 and the transmission/reflection member 59. A concave mirror or the like can be used as the second mirror 55. The image display unit 102 may include a transmissive optical element such as a lens or a prism.

The transmissive/reflecting member 59 is a mirror having a transmittance in the visible range of about 10 to 70%, and a dielectric multilayer film or a wire grid, for example, is formed on the side where the light flux folded by the second mirror 55 enters. Has a reflective surface. The reflection surface of the transflective member 59 can selectively reflect the wavelength band of the light beam emitted by the laser. That is, it may be one having a reflection peak or reflection band that includes light emitted from three lasers corresponding to RGB, or one that is formed so as to increase the reflectance in a specific deflection direction.

The transflective member 59 may be the windshield 125 (see FIG. 1) of the vehicle 120. Further, it may be a member different from the windshield 125. By arranging the image display unit 102 in front of the driver 130 in the vehicle 120, the light flux reflected by the reflection surface of the transflective member 59 enters the eyeball 135 of the driver 130 in the driver's seat. Then, the two-dimensional image of the surface to be scanned 54 is visually recognized by the driver 130 as a virtual image 160 enlarged at a predetermined position in front of the reflecting surface of the transmissive reflecting member 59.

Next, the configuration of the image display system of the vehicle in which the HUD 100 is mounted will be described. FIG. 3A is a block diagram illustrating an example of the configuration of an image display system of a vehicle equipped with a HUD.

The image display system 1 includes a steering angle sensor 61 that communicates via an in-vehicle network NW such as a CAN (Controller Area Network) bus, a seating sensor 62, a vehicle speed sensor 63, a HUD 100, and a wide-angle camera 200.

The steering angle sensor 61 is a sensor that detects the steering angle of the steering wheel by the driver. The steering direction and steering amount are mainly detected. It may be detected by any principle, but for example, there is one that counts ON/OFF of light passing through the slit disk that rotates in conjunction with the steering wheel.

The seating sensor 62 is a sensor that detects whether or not an occupant is seated for each seat of the vehicle. For example, the presence or absence of seating is detected by a pressure detection sensor installed in the seat, an infrared sensor, or the like. It may be detected by a camera that images the interior of the vehicle.

The vehicle speed sensor 63 detects, for example, rotation of wheels by a hall element or the like, and outputs a pulse wave corresponding to the rotation speed. The vehicle speed is detected from the rotation amount (pulse number) per unit time and the outer diameter of the tire.

The HUD 100 can acquire information from each sensor mounted on the vehicle. Further, the HUD 100 may acquire information from an external network instead of the vehicle-mounted network. For example, GPS information, steering angle, vehicle speed, etc. can be acquired. Regarding the steering angle and the vehicle speed, it is considered possible to control the in-vehicle device by observing the position and orientation of the traveling vehicle and the vehicle speed by ITS (Intelligent Transport Systems) when automatic driving is put to practical use in the future.

The wide-angle camera 200 is a camera including a wide-angle lens, and captures a detection image in which the head 131 of the driver 130 in the vehicle 120 and the outside of the vehicle 120 such as the field of view of the driver 130 are included in the same image. You can The details will be described in the first embodiment. Further, instead of the wide-angle camera 200, the stereo camera 300 described in the second embodiment may be connected to the network NW.

Next, the hardware configuration of the processing unit included in the HUD will be described. FIG. 3 is a block diagram showing an example of the hardware configuration of the processing unit according to the embodiment.

The processing unit 101 includes a CPU (Central Processing Unit) 103, a ROM (Read Only Memory) 104, a RAM (Random Access Memory) 105, an input/output I/F (Inter/face) 106, and an SSD (Solid State Drive). ) 107, a GPS (Global Positioning System) receiver 108, an FPGA (Field-Programmable Gate Array), an LD driver 92, and a MEMS controller 93, which are interconnected by a bus B.

The CPU 103 reads a program or data from a storage device such as the ROM 104 or the SSD 107 onto the RAM 105 and executes a process to control the entire processing unit 101 or an arithmetic device that realizes various functions that will be described separately with reference to FIG. Is. Note that some or all of the functions of the CPU 103 may be realized by hardware such as an ASIC (application specific integrated circuit) or FPGA.

The ROM 104 is a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. Programs and data are stored in the ROM 104.

The RAM 105 is a volatile semiconductor memory (storage device) that temporarily holds programs and data. The RAM 105 includes an image memory that temporarily stores image data for the CPU 103 to execute processing such as image processing.

The SSD 107 is a non-volatile storage device that stores programs and data. An HDD (Hard Disk Drive) or the like may be provided instead of the SSD 107.

The input/output I/F 106 is an interface for connecting to an external device such as a wide-angle camera 200 described later. Further, the processing unit 101 can be connected to an in-vehicle network such as a CAN bus via the input/output I/F 106. Further, the processing unit 101 may be connected to an external device such as a PC (Personal Computer) or a video device via the input/output I/F 106.

The GPS receiver 108 is an electric circuit that receives GPS information. The car navigation system may be connected via the CAN bus so that information such as the route and direction can be received from the car navigation system. Further, the car navigation system may realize the functions of the map information storage unit 110 and the current position information acquisition unit 111 described later.

The FPGA 91 controls the LD driver 92 based on the image created by the CPU 103. The LD driver 92 is electrically connected to the light source unit 51, and drives the LD of the light source unit 51 to control the light emission of the LD according to the image.

The FPGA 91 operates the optical deflector 52 by the MEMS controller 93 electrically connected to the optical deflector 52 so that the laser beam is deflected in the direction corresponding to the pixel position of the image.

<Displacement of display image and its position/tilt due to HUD>
Here, a display image by the HUD 100 will be described. FIG. 4 is a diagram illustrating an example of a display image according to the embodiment. (A) is a figure explaining an example of a display image of a mark, (b) is a figure explaining an example of a display image of course information.

In FIG. 4, a field of view of the driver 130 through the windshield 125 (a landscape in front of the vehicle 120, and hereinafter, simply referred to as the field of view of the driver 130) and a display image superimposed and displayed on the field of view. It is shown.

In FIG. 4A, a pedestrian 151 is visually recognized on the road 150, and a mark 141 indicated by a one-dot chain line is superimposed and displayed around the pedestrian 151. By thus displaying the mark 141 so as to be superimposed on the field of view of the driver 130, it is possible to inform the driver 130 of the presence of the pedestrian 151 and call attention.

The position where the mark 141 is displayed in the field of view of the driver 130 can be determined based on the output of the front recognition means such as a stereo camera mounted on the vehicle 120. Further, in this embodiment, as will be described later, at least one of the position and the inclination of the mark 141 (hereinafter, referred to as position/inclination) can be changed according to the image captured by the wide-angle camera 200.

On the other hand, in FIG. 4B, the crossroads 152 are visually recognized, and the route information 142 in which the route is indicated by an arrow is superimposed and displayed on the crossroads 152. In this way, by displaying the route information 142 so as to be superimposed on the field of view of the driver 130, the driver 130 can be prompted to perform a steering wheel operation for turning the vehicle 120 right at the crossroads 152.

The type of the route information 142 such as right turn and left turn can be determined based on the information provided from the GPS or the like. Further, in the present embodiment, as will be described later, the position/inclination of the route information 142 can be changed according to the image captured by the wide-angle camera 200.

Here, the vehicle 120 may cause a change in inclination such as pitching or rolling during traveling due to unevenness of the road surface, braking operation, steering wheel operation, or the like. The view of the driver 130 may change due to the change of the tilt of the vehicle 120, and the position and tilt of the display image by the HUD 100 may be displaced from the view of the driver 130.

FIG. 5 is a diagram illustrating an example of the positional deviation of the display image due to vehicle pitching. 5A is a diagram for explaining the pitching of the vehicle, FIG. 5B is a diagram for explaining a normal display image, and FIG. 5C is a diagram for explaining the positional deviation of the display image due to the pitching.

As shown in FIG. 5A, when the inclination of the vehicle 120 changes in the direction of the thick arrow 121a due to pitching, the field of view of the driver 130 relatively moves in the direction opposite to the thick arrow 121a. Therefore, originally, as shown in FIG. 5B, when the mark 141 is displayed around the pedestrian 151 in front of the vehicle 120, as shown in FIG. 5C, the field of view of the driver 130 is wide. Move in the direction of arrow 121b. As a result, the mark 141 is displaced relative to the pedestrian 151, and is not displayed correctly. In FIG. 5, 126 is a rearview mirror, and 150a is a horizon included in the field of view of the driver 130.

On the other hand, FIG. 6 is a diagram illustrating an example of a tilt shift of a display image due to rolling of the vehicle. FIG. 6A is a diagram for explaining rolling of the vehicle, and FIG. 6B is a diagram for explaining tilt deviation of a display image due to rolling.

As shown in FIG. 6A, when the inclination of the vehicle 120 changes in the direction of the thick arrow 122a due to rolling, the field of view of the driver 130 relatively leans in the direction opposite to the thick arrow 122a. Then, as shown in FIG. 6B, as a result of the driver's field of view tilting in the direction of the thick arrow 122b, the mark 141 causes a relative tilt shift with respect to the pedestrian 151, and the mark is not displayed correctly.

Further, even when the position or posture of the head of the driver 130 changes, the position or inclination of the field of view of the driver 130 changes, so that the above-described shift in the position or inclination of the display image may occur.

FIG. 7 is a diagram illustrating an example of the positional shift of the display image due to the movement of the driver's head. FIG. 7A is a diagram for explaining an image including the driver's head, FIG. 7B is a diagram for explaining movement of the head, and FIG. 7C is a diagram of a display image due to movement of the head. It is a figure explaining a position gap.

In FIG. 7A, the driver 130 is sitting in the driver's seat of the vehicle 120 and is driving the vehicle 120 while visually recognizing the scenery in front of the vehicle 120 through the windshield 125. Here, the scenery in front of the vehicle 120 is an example of “outside the moving body”. However, the “outside of the moving body” is not limited to the scenery in front, and may be behind or to the side of the vehicle 120 as long as the inclination of the vehicle 120 can be detected.

In the case of the state of FIG. 7A, when the head 131 of the driver 130 moves in the direction of the thick arrow 123a as shown in FIG. 7B, the field of view of the driver 130 is relatively thick. Move in the opposite direction. Then, as shown in FIG. 7C, as a result of the driver 130 moving in the direction of the thick arrow 123b, the mark 141 is displaced relative to the pedestrian 151 and is not displayed correctly.

From the above, the image display system 1 according to the present embodiment captures an image in which the head 131 of the driver 130 and the outside of the vehicle 120 are included in the same image (in the same frame. In the same field of view). It has a wide-angle camera 200. The image captured by the wide-angle camera 200 is an example of “an image in which the head of the occupant of the moving body and the outside of the moving body are included in the same image”, and is also an example of the “input image”. Hereinafter, the image captured by the wide-angle camera 200 will be referred to as a detected image.

Based on the image detected by the wide-angle camera 200, the position/tilt of the display image such as the mark 141 and the route information 142 is changed to correct the position/tilt shift. The details will be described below.

[First Embodiment]
<Structure of wide-angle camera>
FIG. 8 is a diagram illustrating an example of the arrangement of the wide-angle camera according to this embodiment in a vehicle. FIG. 8A is a side view of the vehicle, and FIG. 8B is a top view of the vehicle. As shown in FIG. 8, the wide-angle camera 200 is fixed to the central portion of the ceiling inside the vehicle 120.

A room light may be arranged in this portion, but the wide-angle camera 200 may be arranged in the vicinity thereof or may be arranged instead of the room light. Alternatively, the room light and the wide-angle camera 200 may be integrated.

However, the fixed position of the wide-angle camera 200 is not limited to the above-mentioned part, and may be in the vicinity of the room mirror 126 or the like. Alternatively, the room mirror 126 and the wide-angle camera 200 may be integrated.

The wide-angle camera 200 is a camera including a wide-angle lens, and as described above, the head 131 of the driver 130 inside the vehicle 120 and the outside of the vehicle 120 such as the field of view of the driver 130 are detected in the same image. An image can be taken.

However, the present invention is not limited to this, and if the head 131 of the driver 130 and the outside of the vehicle 120 can capture a detection image included in the same image, a wide-angle camera is not necessarily used as the imaging device. Good.

On the other hand, if the image pickup device is a camera having a lens with a viewing angle of 180 degrees or more, even when the image pickup device is arranged near the rearview mirror 126, the head 131 of the driver 130 and the driver 130 in the vehicle 120 can be reduced. The field of view of 130 can be included in the same image. In this case, since the driver 130 is positioned diagonally to the right rear or the left diagonal to the rearview mirror 126, the imaging device is arranged toward the driver 130 in order to include both of them in the visual field. There is a need.

Further, if the image pickup device is a camera having a lens with a viewing angle of 360 degrees, even if the image pickup device is arranged in the vicinity of the room mirror 126, the driver does not need to arrange the image pickup device toward the driver 130. The head 131 of 130 and the landscape in front can be accommodated in the same image.

In addition, by using a camera having a wide-angle lens or an ultra-wide-angle lens as the image pickup device, not only the windshield 125 but also the outside scenery of the vehicle 120 through the side window and the rear window can be seen as the head 131 of the driver 130. Can fit in the same image. Then, these pieces of information can be used for further improving the accuracy of correction of the position/tilt shift of the display image.

Next, the detected image according to this embodiment will be described.

FIG. 9 is a diagram illustrating an example of a detected image according to this embodiment. (A) is a figure explaining the detection image in which the exterior of a vehicle and a driver's head are contained in the same image, and (b) is an enlarged view explaining the details of the exterior of a vehicle.

As shown in FIG. 9A, the detected image includes the head 131 of the driver 130 and the windshield 125. Further, as shown in FIG. 9B, the detected image includes the scenery in front of the driver 130 through the windshield 125.

In the present embodiment, the shift of the display image with respect to the visual field of the driver 130 can be corrected by the functional configuration described below based on such a detected image.

<Functional configuration of the image display system according to the first embodiment>
FIG. 10 is a block diagram illustrating an example of the functional configuration of the image display system according to this embodiment. The functional blocks in FIG. 10 are conceptual and do not necessarily have to be physically configured as illustrated. All or a part of each functional block may be functionally or physically dispersed or combined in arbitrary units.

As shown in FIG. 10, the image display system 1 has a HUD 100 and a wide-angle camera 200. The wide-angle camera 200 also includes an image capturing unit 210 and an image output unit 211. The wide-angle camera 200 has a function of outputting the detection image captured by the image capturing unit 210 to the HUD 100 via the image output unit 211.

The image pickup unit 210 includes a wide-angle lens that forms a subject image, an image pickup device that picks up the formed subject image, and an image pickup control unit that controls the image pickup device. The wide-angle lens can be configured using a plurality of optical elements such as lenses.

A CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor Device), or the like can be used as the image sensor. The image pickup control unit can be realized by an electric circuit that controls exposure of the image pickup element, image reading control, communication with an external circuit, transmission control of image data, and the like. The imaging control unit may be configured to include a CPU, a ROM and the like.

The image output unit 211 has a function of transmitting data and signals to an external device such as the HUD 100, and can be realized by an electric circuit or the like that performs an interface. The image output unit 211 is an example of an “output unit”.

On the other hand, the HUD 100 includes a processing unit 101 and an image display unit 102, and the processing unit 101 includes a detection unit 109, a map information storage unit 110, a current position information acquisition unit 111, and an image generation unit 112. And a changing unit 113.

The detection unit 109 includes a detected image input unit 114, a vehicle tilt detection unit 115, a head position/posture detection unit 116, and an object detection unit 117. The detection image input unit 114 can output the detection image input from the wide-angle camera 200 to the vehicle tilt detection unit 115 and the head position/posture detection unit 116. The detected image input unit 114 can be realized by the input/output I/F 106.

The vehicle tilt detection unit 115 has a function of performing image processing on the input detected image and detecting the tilt of the vehicle 120.

As an example, the vehicle tilt detection unit 115 detects the position (coordinates) and tilt of the horizon 150a (see FIG. 5B, etc.) included in the detected image within the detected image. The position and tilt of the horizon 150a in the detected image relatively means the tilt of the wide-angle camera 200 with respect to the horizon 150a, and since the wide-angle camera 200 is fixed to the vehicle 120, the tilt of the wide-angle camera 200 is the vehicle. It means a slope of 120, ie pitching and rolling. As described with reference to FIGS. 5 and 6, the position of the horizon 150a changes due to the pitching of the vehicle 120, and the inclination of the horizon 150a changes due to the rolling of the vehicle 120.

Therefore, the vehicle inclination detection unit 115 can detect the pitching of the vehicle 120 by detecting the position of the horizon 150a in the detected image, and can detect the rolling of the vehicle 120 by detecting the inclination of the horizon 150a. it can.

As an example, the position of the horizon 150a that hardly changes during traveling indicates a state in which there is no pitching of the vehicle 120, and a change in the position of the horizon 150a in response to this is detected as the pitching of the vehicle 120.

Similarly, the inclination of the horizon 150a that hardly changes during traveling indicates a state in which the vehicle 120 is not rolling, and a change in the inclination of the horizon 150a in response thereto is detected as the rolling of the vehicle 120.

In addition, since the detected image is captured in time series at the imaging cycle (frame rate) of the wide-angle camera 200, the vehicle tilt detection unit 115 causes the vehicle 120 to pitch and roll with respect to the horizon line 150a (hereinafter, in the case where the two are not distinguished from each other, they are also combined. It is possible to detect a change in time (referred to as “tilt”) in accordance with the imaging cycle.

When the detected image does not include the horizon, the vehicle tilt detection unit 115 detects the tilt of the vehicle 120 by using a landscape that is not changed while a distant building or house or a vehicle ahead is running. Good.

Further, the tilt of the running vehicle 120 may be detected based on the optical flow of a subject such as a front vehicle, a pedestrian, or a building included in the detected image, or an image difference between adjacent frames. In this case, a change in the optical flow or the image difference value from the steady state is detected as the inclination of the vehicle 120.

As described above, the position of the scenery outside the vehicle 120 changes due to the pitching of the vehicle 120, and the inclination of the scenery outside the vehicle 120 changes due to the rolling of the vehicle 120. Since the vehicle tilt and the position/tilt of the external scenery are relative as described above, it may be said that the vehicle tilt detecting unit 115 detects the position/tilt of the scenery outside the vehicle 120.

Since a conventional method can be applied to the image processing algorithm for position/tilt detection, description thereof will be omitted here. The vehicle tilt detection unit 115 outputs the detected tilt of the vehicle 120 to the change unit 113.

On the other hand, the head position/orientation detection unit 116 performs image processing on the input detected image to determine at least one of the position and orientation of the head 131 of the driver 130 (hereinafter referred to as position/orientation). It has the function of detecting.

As an example, the position of the head 131 with respect to the horizon 150a is detected, and the posture of the head 131 with respect to the horizon 150a is detected.

Similar to the above, a conventional method can be applied to the image processing algorithm for detecting the position/tilt, and thus the description thereof is omitted here. The head position/orientation detecting unit 116 outputs the detected position/orientation of the head 131 to the changing unit 113.

The object detection unit 117 inputs information on the front of the vehicle 120 from a stereo camera (not shown) and outputs position information of the pedestrian included in the front image to the image generation unit 112. Here, the information in front of the vehicle 120 may be distance information of an object in front of the vehicle 120 or image information in front of the vehicle 120. Further, the distance information of the object in front of the vehicle 120 may be input not only from the stereo camera but also from a device such as a laser radar. On the other hand, the object detection unit 117 may detect the distance information in front of the vehicle 120 by executing parallax calculation processing on the parallax image input from the stereo camera.

The map information storage unit 110 stores map information including road information. Based on the input GPS information, the current position information acquisition unit 111 refers to the map information storage unit 110 to acquire information about the current position of the vehicle 120 and its surroundings, and outputs the information to the image generation unit 112.

The image generation unit 112 generates a display image that displays the mark 141 at a position overlapping the pedestrian included in the field of view of the driver 130, based on the pedestrian position information input from the object detection unit 117.

In addition, the image generation unit 112, based on the current position of the vehicle 120 input from the current position information acquisition unit 111 and the information on the surrounding conditions, the route information 142 at a position overlapping with a road included in the driver's 130 field of view. A display image to be displayed may be generated. The image generating unit 112 outputs the generated display image to the changing unit 113.

The changing unit 113 changes the position/tilt of the display image based on the tilt of the vehicle 120 input from the vehicle tilt detecting unit 115 and the position/posture of the head 131 input from the head position/posture detecting unit 116. .. This makes it possible to correct the deviation of the position/tilt of the display image with respect to the field of view of the driver 130.

Specifically, the changing unit 113 executes geometric transformation image processing for shifting the display image according to the detected inclination of the vehicle 120 due to pitching and/or the position of the head 131. Accordingly, it is possible to correct the positional deviation of the display image with respect to the visual field of the driver 130.

Further, the changing unit 113 executes geometric transformation image processing for rotating the display image according to the detected inclination of the vehicle 120 due to rolling and/or the posture of the head 131. In this case, the center of rotation can be the center of gravity of the mark 141 or the path information 142. This makes it possible to correct the tilt deviation of the display image with respect to the field of view of the driver 130.

However, the correction process by the changing unit 113 is not limited to the geometrically transformed image process of the display image, and is a process of shifting and/or rotating the first mirror and the second mirror included in the image display unit 102. May be.

Note that, in the present embodiment, an example in which the HUD 100 has the function of the processing unit 101 has been described, but the wide-angle camera 200 may include some or all of the functions of the processing unit 101. Further, the processing unit 101 may be provided separately from the HUD 100 and the wide-angle camera 200.

<Operation of Image Display System According to First Embodiment>
Next, the operation of the image display system 1 will be described.

FIG. 11 is a sequence diagram illustrating an example of the operation of the image display system according to this embodiment. Note that, in FIG. 11, a case where the mark 141 is displayed as a display image at a position overlapping with a pedestrian included in the field of view of the driver 130 will be described as an example.

First, in step S111, the object detection unit 117 included in the HUD 100 detects a pedestrian included in a front image of the vehicle 120 input from a stereo camera or the like, and outputs position information of the pedestrian to the image generation unit 112.

Subsequently, in step S112, the image generation unit 112 generates a display image that displays the mark 141 at a position overlapping the pedestrian included in the field of view of the driver 130 based on the input position information of the pedestrian, and the changing unit. Output to 113.

Subsequently, in step S113, the image capturing unit 210 included in the wide-angle camera 200 captures a detected image in which the outside of the vehicle and the head of the driver are included in the same image, and outputs the captured image to the image output unit 211.

Subsequently, in step S114, the image output unit 211 outputs the input detected image to the detected image input unit 114 included in the HUD 100. The detection image input unit 114 outputs the input detection image to the vehicle tilt detection unit 115 and the head position/posture detection unit 116.

Note that the order of the display image generation processing in steps S111 to S112 and the detection image capturing and output processing in steps S113 to S114 can be changed as appropriate, and the display image generation processing and the detection image capturing and output processing can be changed. It may be executed in parallel.

Subsequently, in step S115, the vehicle tilt detection unit 115 detects the tilt of the vehicle 120 based on the input detected image, and outputs the detection result to the change unit 113.

Subsequently, in step S116, the head position/orientation detecting unit 116 detects the position/orientation of the head 131 based on the input detected image, and outputs the detection result to the changing unit 113.

The order of the processes of steps S115 to S116 can be changed as appropriate, and the processes of S115 to S116 may be executed in parallel.

Subsequently, in step S117, the changing unit 113 changes the position/tilt of the mark 141 according to the input tilt of the vehicle 120 and/or the position/orientation of the head 131. Then, the changing unit 113 outputs the display image in which the position/tilt of the mark 141 is changed to the image display unit 102.

Subsequently, in step S118, the image display unit 102 displays the input display image.

In this way, the image display system 1 can visually display the display image, in which the displacement of the position/tilt is corrected, to the driver 130 through the windshield 125.

In the above description, the HUD 100 includes the object detection unit 117, but an external device connected to the CAN bus may have the function of the object detection unit 117. In this case, the process of step S111 is executed by the external device, and the pedestrian position information is input to the image generation unit 112. The processing after step S111 is the same as that described above.

Further, in the above description, the case where the mark 141 is displayed as the display image has been described as an example, but the position/tilt deviation of the route information 142 may be corrected and displayed. In this case, in step S111, the current position information acquisition unit 111 included in the HUD 100 acquires the current position of the vehicle 120 and information about the surrounding conditions thereof by referring to the map information storage unit 110 based on the input GPS information. , To the image generation unit 112. The process after step S112 is the same as that described above except that the display image becomes the route information 142.

<Effect>
Next, operation effects of the image display system 1 according to the present embodiment will be described.

FIG. 12 is a diagram illustrating an example of the display image after the position and the tilt shift are corrected. (A) is a figure explaining the display image after position shift correction by pitching of a vehicle, (b) is a figure explaining the display image after inclination shift correction by rolling of a vehicle, (c) is a figure of a passenger. It is a figure explaining the display image after position gap correction by movement of a head.

As shown in FIG. 12, according to the present embodiment, the position and the inclination of the mark 141 can be changed according to the change in the field of view of the driver 130, and the mark 141 can be appropriately displayed around the pedestrian 151.

On the other hand, in the above description, an example of the vertical displacement of the display image due to pitching has been described. However, in the present embodiment, the pedestrian is on the slope of the slope before the vehicle 120 approaches the slope of the slope. Is also effective in correcting the positional deviation of the display image in the vertical direction.

FIG. 13 is a diagram illustrating a first example of the effect of correcting the positional deviation of the display image due to such a slope. FIG. 13A is a diagram for explaining the case of an uphill, and FIG. 13B is a diagram for explaining the case of a downhill.

In FIG. 13( a ), the vehicle 120 is traveling on a road 150 having no slope and is in a state before approaching the uphill 153 in the front. A pedestrian 151 exists on the slope of the uphill 153.

In this case, since the vehicle 120 is not tilted, the front recognition means such as a stereo camera included in the vehicle 120 recognizes that the pedestrian 151 is present on the extension line of the road 150 which is not tilted. Therefore, a vertical displacement may occur between the display image that is displayed assuming the road 150 having no slope and the pedestrian 151 that actually exists on the slope.

In the case of vertical displacement due to pitching, for example, the position of the display image can be corrected using the output of a posture sensor such as a gyro sensor provided in the vehicle. However, in the case shown in FIG. 13, since the vehicle 120 itself is not leaning, it is not possible to appropriately correct the positional deviation of the display image by the output of the attitude sensor. The same applies to the case of the downhill shown in FIG.

On the other hand, in the present embodiment, since the positional deviation of the display image is corrected according to the detected image including the outside of the vehicle 120, the mark around the pedestrian 151 included in the field of view of the driver 130 is appropriately marked. 141 can be displayed.

Next, FIG. 14 is a diagram illustrating a second example of the effect of correcting the positional deviation of the display image due to the slope. In FIG. 14, the vehicle 120 is traveling on an uphill 153 having an inclination θ and is inclined at a pitching angle θa with respect to a road on a slope.

In this case, as a comparative example, when the display position of the display image is corrected according to the output of the attitude sensor such as a gyro sensor provided in the vehicle, the output of the attitude sensor is based on the vertical direction. It is not possible to appropriately correct the positional deviation of the displayed image that is output and that is caused by the pitching angle θa with respect to the road on the slope.

On the other hand, in the present embodiment, since the positional deviation of the display image is corrected according to the detected image including the outside of the vehicle 120, the positional deviation of the display image due to the pitching angle θa is detected and corrected. Therefore, the mark 141 can be appropriately displayed around the pedestrian 151 included in the field of view of the driver 130.

Next, FIG. 15 is a diagram illustrating a first example of the positional deviation correction effect of the display image of the route information, (a) is a diagram illustrating a display image of the route information before correction, and (b). FIG. 6 is a diagram illustrating a display image of route information after correction.

FIG. 15 shows an example in which there is a crossroad in front of the running vehicle 120 and the route information for prompting a right turn at the crossroad is displayed.

The field of view of the driver 130 changes as described above according to the inclination of the vehicle 120 and/or the position/orientation of the head 131. On the other hand, the GPS information that provides the course information does not include information about the inclination of the vehicle 120 and the position/orientation of the head 131. Therefore, a deviation occurs between the two, and the route information is displayed at a position before the intersection road 155 as shown in the route information 143 of FIG. 15A, or the intersection information 155 is displayed as the route information 144. There is a case where tilted course information is displayed.

On the other hand, in the present embodiment, since the positional deviation of the display image is corrected based on the detected image in which the head 131 and the outside of the vehicle 120 are included in the same image, as shown in FIG. The route information 142 can be displayed appropriately.

Next, FIG. 16 is a diagram illustrating a second example of the positional deviation correction effect of the display image of the route information. (A) is a figure explaining a slope, (b) is a figure explaining a display image of course information after position gap amendment by a slope.

In FIG. 16( a ), the vehicle 120 is traveling on the sloping road 150 and is approaching the downhill 154. An intersection road 155 exists in the middle of the downhill 154.

In this case, since the vehicle 120 itself is not tilted, the front recognition means included in the vehicle 120 recognizes that the intersection road 155 is located on the extension of the road 150 having no slope. Then, there may be a vertical displacement between the route information displayed on the assumption that the road 150 has no slope and the intersection road 155 that actually exists on the slope.

In the present embodiment, since the positional deviation of the display image is corrected based on the detected image including the outside of the vehicle 120, even in such a case, as shown in FIG. The route information can be properly displayed at the position.

As described above, in the present embodiment, the position/tilt of the display image is changed according to the detected image in which the head 131 of the driver 130 and the outside of the vehicle 120 are included in the same image. As a result, it is possible to appropriately correct the tilt of the vehicle 120 due to pitching, rolling, or the like, or the position/tilt shift of the display image due to the position/orientation of the head 131 of the driver 130.

Further, the tilt of the vehicle 120 and the position/orientation of the head 131 are detected based on the information included in the same image. This makes it unnecessary to calibrate the mutual relationship of the outputs, which is necessary when the tilt of the vehicle 120 and the position/orientation of the head 131 are detected by different constituent elements, and expands without the need for calibration. A realistic image display can be performed.

<Modification>
Here, an image display system according to a modified example of the first embodiment will be described.

In the first embodiment, the position/tilt of the wide-angle camera 200 is considered to mean the position/tilt of the vehicle 120, and an example of detecting the position/tilt of the vehicle 120 with respect to the field of view of the driver 130 has been shown. In this case, if the position/tilt of the wide-angle camera 200 fixed to the vehicle 120 changes with time, a detection error may occur. In order to prevent such a detection error, it is necessary to calibrate the position/tilt relationship between the wide-angle camera 200 and the vehicle 120, which may be troublesome.

Therefore, in the modification, the calibration of the wide-angle camera 200 is performed by including the internal structure of the vehicle 120 in the same image and detecting the position and inclination of the internal structure of the vehicle 120 with respect to the outside. The position/inclination of the vehicle 120 with respect to the outside can be detected without any trouble.

FIG. 17 is a diagram illustrating an example of a detection method by the image display system according to the modification. (A) is a figure explaining a normal display image, (b) is a figure explaining the gap of the position and inclination of the detected display image.

In FIG. 17, as an example of the internal structure of the vehicle 120, the upper side 127 of the frame of the windshield 125 is used, and the position/tilt outside the vehicle 120 is detected based on the upper side 127. In FIG. 17A, the horizon 150a is located at a position separated by P downward with respect to the upper side 127, and the inclination is zero.

On the other hand, in FIG. 17B, the horizon line 150a is located at a position separated by P+s downward with respect to the upper side 127, and a shift s occurs in the vertical direction with respect to FIG. 17A. Further, the horizon 150a is inclined by the angle A with respect to the upper side 127.

In this way, the position/tilt outside the vehicle 120 can be detected based on the internal structure of the vehicle 120.

Since the internal structure of the vehicle 120 is used as a reference, even if the position/tilt of the wide-angle camera 200 with respect to the vehicle 120 changes with time, no detection error occurs. Therefore, the position/tilt of the vehicle 120 with respect to the outside can be detected without taking the trouble of calibration.

Although the upper side 127 of the frame of the windshield 125 has been described as an example of the internal structure, the invention is not limited to this, and the room mirror 126 or the like may be used.

[Second Embodiment]
Next, an image display system according to the second embodiment will be described. Note that description of the same components as those of the above-described embodiment will be omitted.

<Structure of Image Display System According to Second Embodiment>
FIG. 18A is a diagram illustrating an example of the configuration of the image display system according to the present embodiment. The image display system 1a has a HUD 100a and a stereo camera 300 as an example of an imaging device. As shown in FIG. 18A, stereo camera 300 is fixed to the central portion of the ceiling inside vehicle 120.

Stereo camera 300 can capture a detection image in which the head 131 of driver 130 of vehicle 120 and the outside of vehicle 120 are included in the same image. The image display system 1a can acquire the three-dimensional position information of the head 131 of the driver 130 based on the image detected by the stereo camera 300, and correct the position/tilt deviation of the display image.

<Functional configuration of image display system according to second embodiment>
FIG. 18B is a block diagram illustrating an example of the functional configuration of the image display system according to this embodiment. As shown in FIG. 18B, the image display system 1a has a HUD 100a and a stereo camera 300. In addition, the stereo camera 300 includes a first imaging unit 301, a second imaging unit 302, a detection image output unit 303, a parallax image acquisition unit 304, a distance image acquisition unit 305, and a distance image output unit 306. The HUD 100 has a distance image input unit 118, a head position/orientation detection unit 116a, an object detection unit 117a, and a change unit 113a.

The first imaging unit 301 includes a lens that forms a subject image, an imaging device that captures the formed subject image, and an imaging control unit that controls the imaging device, and the head of the driver 130 of the vehicle 120. The detection image in which the unit 131 and the outside of the vehicle 120 are included in the same image is captured. The configuration of the second image pickup unit 302 is similar to that of the first image pickup unit 301.

The first imaging unit 301 is provided at a position separated from the second imaging unit 302 by the baseline length, and the detected images captured by each include a parallax corresponding to the baseline length. Here, FIG. 19 is a diagram illustrating an example of an image detected by the stereo camera according to the present embodiment. (A) is a figure explaining the detection image by a 1st imaging part, (b) is a figure explaining the detection image by a 2nd imaging part.

As shown in FIG. 19, there is a parallax in the left-right direction between the image detected by the first imaging unit 301 and the image detected by the second imaging unit 302. Based on this parallax, the stereo camera 300 can detect the distance to the head 131 of the driver 130. For convenience of description, the image detected by the first imaging unit 301 will be referred to as a first detection image, and the image detected by the second imaging unit 302 will be referred to as a second detection image hereinafter.

Returning to FIG. 18B, the description will be continued. The first imaging unit 301 outputs the first detection image to the detection image output unit 303 and the parallax image acquisition unit 304, and the second imaging unit 302 outputs the second detection image to the parallax image acquisition unit 304.

The detection image output unit 303 outputs the input first detection image to the detection image input unit 114 of the HUD 100a. The detection image output unit 303 may input the second detection image and output the second detection image to the detection image input unit 114.

The parallax image acquisition unit 304 calculates the parallax between the first detection image and the second detection image that have been input, and acquires the parallax image. In the parallax calculation processing, block matching image processing or the like is executed, and a parallax image or the like including parallax for each minute block of the detected image is acquired. Since a known technique can be applied to the parallax calculation process, the description thereof is omitted here. The parallax image acquisition unit 304 outputs the acquired parallax image to the distance image acquisition unit 305.

The distance image acquisition unit 305 converts the parallax of the parallax image into a distance, acquires the distance image, and outputs the distance image to the distance image output unit 306. The distance image output unit 306 outputs the input distance image to the distance image input unit 118 of the HUD 100a.

The head position/orientation detection unit 116a extracts the head 131 of the driver 130 from the distance image input via the distance image input unit 118, detects the three-dimensional position/orientation of the head 131, and performs detection. The result is output to the changing unit 113a.

The object detection unit 117a can detect a pedestrian included in the distance information ahead of the vehicle 120 input from the stereo camera 300, and output the pedestrian position information to the image generation unit 112. However, the object detection unit 117a may input image information from the stereo camera 300, detect a pedestrian included in the image information, and output position information of the pedestrian to the image generation unit 112.

Further, the distance information in front of the vehicle 120 or the image information may be input to the object detection unit 117a from an external device connected to the CAN bus.

The changing unit 113a changes the position/tilt of the display image based on the input three-dimensional position/orientation of the head 131. This corrects the positional deviation of the position/tilt of the display image with respect to the visual field of the driver 130.

Note that, in the present embodiment, an example in which the HUD 100a has the processing unit 101a has been described, but the stereo camera 300 may have some or all of the functions of the processing unit 101a. Further, the processing unit 101a may be provided separately from the HUD 100a and the stereo camera 300.

<Operation of Image Display System According to Second Embodiment>
Next, the operation of the image display system 1a will be described.

FIG. 20 is a sequence diagram illustrating an example of the operation of the image display system according to this embodiment. In addition, in FIG. 20, a case where the mark 141 is displayed as a display image at a position overlapping with a pedestrian included in the field of view of the driver 130 will be described as an example.

First, in step S201, the first imaging unit 301 included in the stereo camera 300 captures a first detection image and outputs the first detection image to the detection image output unit 303 and the parallax image acquisition unit 304.

Subsequently, in step S202, the second imaging unit 302 images the second detection image and outputs the second detection image to the parallax image acquisition unit 304.

Subsequently, in step S203, the parallax image acquisition unit 304 calculates the parallax between the first detection image and the second detection image that have been input, and acquires the parallax image. The parallax image acquisition unit 304 outputs the acquired parallax image to the distance image acquisition unit 305.

Subsequently, in step S204, the distance image acquisition unit 305 converts the parallax of the parallax image into a distance, acquires the distance image, and outputs the distance image to the distance image output unit 306.

Subsequently, in step S205, the detection image output unit 303 outputs the input detection image to the detection image input unit 114 included in the HUD 100, and the detection image input unit 114 outputs the input detection image to the vehicle tilt detection unit 115. To do.

Subsequently, in step S206, the distance image output unit 306 outputs the input distance image to the distance image input unit 118, and the distance image input unit 118 outputs the input distance image to the head position/posture detection unit 116a. To do.

Subsequently, in step S207, the object detection unit 117a included in the HUD 100a detects a pedestrian included in the distance image input from the stereo camera 300, and outputs position information of the pedestrian to the image generation unit 112.

Subsequently, in step S208, the image generation unit 112 generates a display image that displays the mark 141 at a position overlapping the pedestrian included in the field of view of the driver 130 based on the input position information of the pedestrian, and the changing unit. It is output to 113a.

The order of the distance image acquisition process of steps S201 to S204 and the display image generation process of steps S207 to S208 can be changed as appropriate, and the distance image acquisition process and the display image generation process may be executed in parallel. .. Similarly, the order of the processes of steps S205 to S206 can be changed as appropriate, and the processes of S205 to S206 may be executed in parallel.

Subsequently, in step S209, the vehicle tilt detection unit 115 detects the tilt of the vehicle 120 based on the input first detection image, and outputs the detection result to the change unit 113a.

Subsequently, in step S210, the head position/orientation detecting unit 116 detects the three-dimensional position/orientation of the head 131 based on the input distance image, and outputs the detection result to the changing unit 113a.

When performing the processing of S207 to S208 after the distance image acquisition processing, the order of the processing of steps S207 to S210 can be changed as appropriate, and the processing of S207 to S210 may be executed in parallel.

Subsequently, in step S211, the changing unit 113a changes the position/tilt of the mark 141 according to the input tilt of the vehicle 120 and the position/orientation of the head 131. Then, the changing unit 113a outputs the display image in which the position/tilt of the mark 141 is changed to the image display unit 102.

Subsequently, in step S212, the image display unit 102 displays the input display image.

In this way, the image display system 1a can visually display the display image, in which the displacement of the position/tilt is corrected, to the driver 130 through the windshield 125.

Note that, in the above description, the case where the mark 141 is displayed as a display image has been described as an example, but the position/tilt deviation of the route information 142 may be corrected and displayed. In this case, in step S201, the current position information acquisition unit 111 included in the HUD 100a refers to the map information storage unit 110 based on the input GPS information, and acquires the current position of the vehicle 120 and information on the surrounding conditions thereof. , To the image generation unit 112. The process after step S202 is the same as that described above except that the display image becomes the route information 142.

As described above, in the present embodiment, the stereo camera 300 is used as an imaging device and the three-dimensional position information of the head 131 of the driver 130 is used. As a result, the position/orientation of the head 131 can be detected with high accuracy, and the deviation in the position/tilt of the display image due to the change in the position/orientation of the head 131 can be more appropriately corrected.

The other effects are similar to those described in the above embodiment.

[Third Embodiment]
Next, an image display system according to the third embodiment will be described.

The image display system according to the present embodiment displays a display image including a passive display image and an active display image. Here, the passive display image is an image whose display position/tilt is changed by the changing unit 113, and the active display image is an image whose display position/tilt is not changed by the changing unit 113. Is.

FIG. 21 is a diagram illustrating an example of a display image according to this embodiment. As shown in FIG. 21, a driver's view 221 through the windshield includes a front vehicle 222. Further, a passive display image 223 and an active display image 224 are displayed so as to be superimposed on the driver's visual field 221.

The passive display image 223 is an image that displays an arrow indicating the course of the vehicle 120 that is the host vehicle, and the position/tilt is changed according to the tilt of the vehicle 120 and the position/posture of the head 131 of the driver 130. Image.

On the other hand, as shown in FIG. 21, the active display image 224 is an image displaying the traveling speed “30 km” and the distance “100 m” to the right turn point. Since these present information that is irrelevant to the field of view of the driver 130, it is not necessary to change the position/tilt according to the change in the field of view of the driver 130, but instead, the information is displayed in a stationary state. Better information is better. Therefore, in this embodiment, the active display image 224 is an image in which neither the display position nor the inclination is changed by the changing unit 113.

As described above, according to the present embodiment, only the position/tilt of necessary information is changed according to the change in the field of view of the driver 130, so that the image display with augmented reality can be appropriately performed.

Although the preferred embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and replacements of the above-described embodiments are possible without departing from the scope of the claims. Can be added.

Specifically, the moving body may be a two-wheeled vehicle, a four-wheeled vehicle such as a general vehicle, a construction/agriculture/industrial vehicle, a railway vehicle, a special vehicle, or a flying body such as a drone. Good. These are collectively called a moving body.

The embodiment also includes an image display method. For example, the image display method is an image display method using an image display system provided in a moving body, which includes an image pickup device and an image display device, wherein the image pickup device is a head of an occupant of the moving body. A step of outputting the image to the image display device, and an image capturing step of capturing an image included in the same image outside the moving body, and an output step of outputting the image to the image display device. And an image display step of displaying the generated display image at a position overlapping the outside of the moving body when visually recognized by the occupant, and displaying the display image according to the image. And a changing step of changing at least one of the position and the inclination.

With such an image display method, it is possible to obtain the same effect as the above-described image display system. Further, such an image display method may be realized by a circuit such as a CPU or an LSI, an IC card or a single module.

1, 1a Image display system 51 Light source unit 52 Optical deflector 53 First mirror 54 Scanned surface 55 Second mirror 59 Transmission/reflection member 61 Steering angle sensor 62 Seating sensor 63 Vehicle speed sensor 91 FPGA
92 LD driver 93 MEMS controller 100, 100a HUD (an example of image display device)
101, 101a processing unit 102 image display unit 103 CPU
104 ROM
105 RAM
106 Input/output I/F
107 SSD
108 GPS receivers 109, 109a Detection unit 110 Map information storage unit 111 Current position information acquisition unit 112 Image generation unit 113, 113a Change unit 114 Detection image input unit 115 Vehicle tilt detection unit 116, 116a Head position/posture detection unit 117 117a Object detection unit 118 Distance image input unit 120 Vehicle (an example of a moving body)
125 Windshield 126 Interior mirror 127 Upper side (an example of internal structure)
130 Driver (an example of a passenger)
131 Head 135 Eyeball 141 Mark 142-144 Path information 150 Road 150a Horizon 151 Pedestrian 152 Crossroad 153 Uphill 154 Downhill 155 Crossroad 160 Virtual image 200 Wide-angle camera 210 Imaging unit 211 Image output unit (an example of output unit)
221 Field of view 222 Forward vehicle 223 Passive display image 224 Active display image 300 Stereo camera 301 First imaging unit 302 Second imaging unit 303 Detection image output unit 304 Parallax image acquisition unit 305 Distance image acquisition unit 306 Distance image output unit

JP, 2017-087826, A

Claims (11)

An image display system having an image pickup device and an image display device, the image display system being provided on a moving body,
The imaging device is
A head of the occupant of the moving body, and the outside of the moving body, and an imaging unit that captures images included in the same image,
The image display device,
An image generation unit that generates a display image,
An image display unit that displays the generated display image at a position overlapping the outside of the moving body when visually recognized by the occupant,
An image display system comprising: a changing unit that changes at least one of a position and an inclination at which the display image is displayed according to the image. A detection unit that detects at least one of the position of the head of the occupant, the posture of the head of the occupant, and the inclination of the moving body based on the image;
The image display system according to claim 1. In the image, in addition to the head of the occupant of the moving body and the outside of the moving body, the internal structure of the moving body is further included in the same image,
The image display system according to claim 1, wherein the detection unit detects an inclination of the moving body with respect to the outside based on at least one of a position and an inclination of the internal structure with respect to the outside. The image display system according to claim 1, wherein the imaging device has a wide-angle lens. The image display system according to claim 1, wherein the imaging device includes a stereo camera. The image display system according to claim 1, wherein the display image includes information indicating a course of the moving body. In the display image,
At least one of a position and a tilt at which the display image is displayed is changed by the changing unit, and
The image display system according to any one of claims 1 to 6, further comprising: an active display image whose position and inclination for displaying the display image are not changed by the changing unit. A moving body comprising the image display system according to claim 1. An image display method using an image display system provided on a moving body, the method including an imaging device and an image display device,
The imaging device is
An imaging step of capturing an image in which the head of the occupant of the moving body and the outside of the moving body are included in the same image,
An output step of outputting the image to the image display device,
The image display device,
An image generation process for generating a display image,
An image display step of displaying the generated display image at a position overlapping the outside of the moving body when visually recognized by the occupant,
And a changing step of changing at least one of a position and a tilt at which the display image is displayed according to the image. An image display device provided on a moving body,
An image generation unit that generates a display image,
An image display unit that displays the generated display image at a position overlapping the outside of the moving body when visually recognized by an occupant,
A changing unit that changes at least one of a position and an inclination at which the display image is displayed based on an input image in which the head of the occupant of the moving body and the outside of the moving body are included in the same image. And an image display device having. In the computer of the image display device provided in the moving body,
An image generation step of generating a display image,
An image display step of displaying the generated display image at a position overlapping the outside of the moving body when visually recognized by an occupant,
A changing step of changing at least one of a position and an inclination at which the display image is displayed, based on an input image in which the head of the occupant of the moving body and the outside of the moving body are included in the same image. And a program to execute.

Images