JP2021105986A - Display device, moving entity, display method, and program - Google Patents

Abstract

To indicate a path to which a traffic sign is applied or the traffic sign despite a narrow projection angle of view.SOLUTION: A display device according to an embodiment of the present invention displays a virtual image of an image to be projected, and includes: an image production unit that produces the image to be projected which contains a predetermined image acquired based on a traffic sign; and a direction instruction unit that outputs an existing direction in which either an object path to which the traffic sign is applied or the traffic sign exists, on the basis of a position of the traffic sign and a position of the path ahead of a moving entity, and allows the image production unit to produce the image to be projected containing the existing direction.SELECTED DRAWING: Figure 11

Description

The present invention relates to a display device, a moving body, a display method and a program.

Conventionally, a display device such as a head-up display has been known in which a virtual image of a projected image including various information is superimposed and displayed on the foreground of the vehicle on the front windshield of the vehicle or the like so that the driver of the vehicle can visually recognize the virtual image.

In addition, it recognizes traffic signs such as road signs and road surface signs in front of the vehicle based on the front image of the vehicle acquired by the camera attached to the vehicle, and displays a virtual image of the virtual sign image at the position of the road visible in the foreground. A technique for instructing a vehicle driver on a road to which a traffic sign is applied by superimposing a display is disclosed (see, for example, Patent Document 1).

However, in the technique of Patent Document 1, when a route such as a road to which a traffic sign is applied or a traffic sign is visually recognized near the edge of the foreground, a virtual sign image or the like is specified at the position of the traffic sign in the foreground. In order to superimpose and display a virtual image of an image, a wide-angle projection angle of view is required for the display device, which may increase the size and cost of the display device.

An object of the present invention is to show a route or a traffic sign to which a traffic sign is applied even with a narrow projection angle of view.

The display device according to one aspect of the present invention is a display device that displays a virtual image of a projected image, and includes an image generation unit that generates the projected image including a predetermined image acquired based on a traffic sign, and the above. Based on the position of the traffic sign and the position of the route in front of the moving body, the existence direction in which either the target route to which the traffic sign is applied or the traffic sign exists is output, and the existence direction is determined. It includes a direction indicating unit that causes the image generation unit to generate the projected image including the image.

Even with a narrow projection angle of view, it is possible to indicate the route or traffic sign to which the traffic sign is applied.

It is a figure which shows the configuration example in which the display device which concerns on embodiment is mounted on a vehicle. It is a figure which shows the structural example of the projection part. It is a block diagram which shows the configuration example of the in-vehicle system in which the display device is mounted on a vehicle. It is a block diagram which shows the configuration example of a detection device. It is a block diagram which shows the functional structure example of an image processing apparatus. It is a block diagram of the hardware configuration example of the control part which concerns on embodiment. It is a block diagram of the functional structure example of the control part which concerns on 1st Embodiment. It is a flowchart which shows the processing example by the control part which concerns on 1st Embodiment. It is a figure which shows an example of the front image. It is a figure which shows an example of the projected image. It is a figure which shows an example of the foreground of the vehicle which concerns on 1st Embodiment. It is a figure which shows an example of the foreground which changed from the state of FIG. 11 by the movement of a vehicle. It is a figure explaining the wide-angle of a projection part. It is a figure which shows the functional structure example of the control part which concerns on 2nd Embodiment. It is a flowchart which shows the processing example of the control part which concerns on 2nd Embodiment. It is a figure which shows an example of the foreground of the vehicle which concerns on 2nd Embodiment. It is a figure which shows an example of the foreground which changed from the state of FIG. 16 by the movement of a vehicle.

Hereinafter, modes for carrying out the invention will be described with reference to the drawings. In each drawing, parts having the same configuration may be designated by the same reference numerals, and duplicate description may be omitted.

In the embodiment, the display device superimposes and displays a virtual image of a projected image including various information on a front windshield (windshield) or the like, and makes it visible to a driver of a moving body such as a vehicle.

In addition, a projected image including a predetermined image such as a virtual sign image acquired based on the traffic sign is generated, and the traffic sign is applied based on the position of the traffic sign and the position of the route in front of the moving body. Show the driver the direction in which either the target route or the traffic sign is present.

By indicating the direction, it is not necessary to widen the projection angle of view for superimposing and displaying a virtual image of a predetermined image such as a virtual sign image on the position of the target path visually recognized in the foreground. This makes it possible to show either the target route or the traffic sign even if the projection angle of view of the display device is narrow.

<Terms in the embodiment>
(image)
Both still images and moving images are included, and video is also included in the image.

(foreground)
The scenery in front of the moving body that the driver of the moving body sees. For example, the foreground is the scenery in front of the driver as seen through the front windshield of the vehicle.

(driver)
A person who drives or steers a moving body. For example, a vehicle driver and the like can be mentioned.

(Traffic sign)
A product that provides necessary information to those who use transportation. For example, a traffic sign includes a display member which is a member for displaying information, an image for displaying information, a paint for displaying information, and the like. More specifically, the traffic sign includes a road sign, a road marking, and the like. A road sign is a display board provided near or above the road and providing necessary information to a person who uses the road, such as a vehicle driver. Road markings are paints on the road surface to provide information for the purpose of organizing, guiding or regulating traffic. Traffic signs indicate predetermined figures, symbols, characters, etc. according to the information to be provided.

(Mobile)
A movable object. For example, moving objects include land-movable land-moving bodies such as automobiles, vehicles, trains, trains or forklifts, and air-moving aerial-moving bodies such as airplanes, balloons or drones, including ships, ships and steamers. Alternatively, it includes a water-moving body such as a boat that can move on the water.

(route)
A road, road, way or direction in which a moving body moves. For example, the route includes a traffic route, a traveling direction, a traveling direction, etc., includes a road, a passage, a passage, a traveling route, a railroad, etc. , Includes waterways, sea routes, sea routes, etc. to which water moving objects move.

(Virtual sign image)
An image of a virtual traffic sign, which means a figure corresponding to a figure displayed by a traffic sign or an image displaying at least one of characters corresponding to a character displayed by a traffic sign. For example, when the traffic sign is a "no entry" road sign, an image having the same pattern as the pattern displayed by the "no entry" road sign corresponds to the virtual sign image.

(Projected image)
An image projected for the driver to see. The projected image includes information to be provided to the driver, such as vehicle traveling speed information and virtual sign images.

(Predetermined image)
An image included in the above-mentioned projected image. For example, the above-mentioned virtual sign image, a character image indicating a character in which a traffic sign is replaced with a word, or the like corresponds to a predetermined image.

(Predetermined position)
It refers to the position where the above-mentioned predetermined image is placed in the projected image when the traffic sign is removed from the foreground due to the movement of the moving body and the traffic sign information is no longer acquired.

(Target route)
The route to which the information indicated by the traffic sign is applied. For example, when a traffic sign indicates that a speed limit of 60 km / h is applied to a road on which an automobile travels, the road to which the speed limit of 60 km / h is applied corresponds to the target route. The road in this case is called the target road. The target road is an example of the target route.

(Direction of existence)
The direction in which either the target route or the traffic sign exists when viewed from the driver.

Hereinafter, embodiments will be described using a head-up display (HUD) mounted on a vehicle that can move on the road as an example. Here, the road is an example of a route, the vehicle is an example of a moving body, and the HUD is an example of a display device.

<Configuration example>
FIG. 1 is a diagram showing an example of a configuration in which the display device 1 according to the embodiment is mounted on the vehicle 8. As shown in FIG. 1, the display device 1 includes a control unit 20 and a projection unit 10.

The display device 1 is embedded inside a dashboard provided in the vehicle 8, and projects an image from an injection window 3 provided on the upper surface of the display device 1 toward the front windshield 71. The projected image is displayed as a virtual image I in the positive Y direction from the front windshield 71, and is visually recognized by the driver V of the vehicle 8. By visually recognizing the virtual image I, the driver V can acquire useful information for driving with a small movement of the line of sight while keeping the line of sight of another vehicle or the road surface in the positive Y direction of the vehicle 8.

The display device 1 may be installed on the ceiling, sun visor, or the like in addition to the dashboard, as long as the image can be projected on the front windshield 71.

As shown in FIG. 1, the display device 1 includes a projection unit 10 and a control unit 20 as main components. As the projection method of the projection unit 10, a laser scanning method and a panel method are known. The laser scanning method is a method in which a laser beam emitted from a laser light source is scanned by a two-dimensional scanning device to form an intermediate image (a real image projected on a screen described later). The panel method is a method of forming an intermediate image with an imaging device such as a liquid crystal panel, a DMD (Digital Micro-mirror Device) panel, or a vacuum fluorescent display (VFD). The projected image includes both those projected by the laser scanning method and those projected by the panel method.

The laser scanning method is different from the panel method in which an image is formed by emitting light from the entire screen and partially blocking light, and since it is possible to assign light emission / non-light emission to each pixel, a high-contrast image is generally obtained. It is suitable in that it can be formed. Since the high contrast improves the visibility, it has been clarified that the occupants of the vehicle can visually recognize the information with less attention resources than the panel type HUD.

In particular, in the panel method, light that cannot be completely blocked is projected even in an area where there is no information, and the display frame is projected in a range where the HUD can display an image (this phenomenon is called a postcard). The laser scanning method does not have such a phenomenon and can project only the content. In particular, in AR (Augmented Reality), the reality when the generated image is superimposed and displayed on the actual landscape is improved.

Note that AR is translated as "augmented reality" and refers to a technology that virtually expands the world in front of you by superimposing images of non-existent objects on an existing landscape. However, even if it is a panel-type HUD, it may be a HUD that can display information in a manner that can be visually recognized (less tired) with less attention resources.

FIG. 2 is a diagram showing an example of the configuration of the projection unit 10. The projection unit 10 mainly includes a light source unit 101, a light deflector 102, a mirror 103, a screen 104, and a concave mirror 105. Note that FIG. 2 only shows the main components, and may include components other than those shown in the figure, or may not include some of the components shown in the figure.

The light source unit 101 includes three laser light sources (hereinafter referred to as LD: laser diode) corresponding to Red (R), Green (G), and Blue (B), a coupling lens, an aperture, a synthetic element, a lens, and the like. The laser beams emitted from the three LDs are combined and guided toward the reflecting surface of the light deflector 102. The laser beam guided to the reflecting surface of the light deflector 102 is two-dimensionally deflected by the light deflector 102.

As the optical deflector 102, one minute mirror that swings with respect to two orthogonal axes, two minute mirrors that swing or rotate about one axis, and the like can be used. The optical deflector 102 can be a MEMS (Micro Electro Mechanical Systems) mirror manufactured by a semiconductor process or the like. The optical deflector 102 can be driven by an actuator whose driving force is the deformation force of the piezoelectric element. However, as the light deflector 102, a galvano mirror, a polygon mirror, or the like may be used.

The laser beam two-dimensionally deflected by the light deflector 102 is incident on the mirror 103, folded back by the mirror 103, and draws a two-dimensional image (intermediate image) on the surface (scanned surface) of the screen 104. As the mirror 103, a concave mirror or the like can be used, but a convex mirror or a plane mirror may also be used. By deflecting the direction of the laser beam with the light deflector 102 and the mirror 103, the size of the projection unit 10 can be reduced or the arrangement of the components can be flexibly changed.

As the screen 104, it is preferable to use a microlens array or a micromirror array having a function of diverging the laser beam at a desired divergence angle, but a diffuser plate for diffusing the laser beam, a transmission plate having a smooth surface, or a reflector. Etc. may be used. Generally, the light source unit 101 to the screen 104 are called a HUD device. However, other parts may be included.

The laser beam emitted from the screen 104 is reflected by the concave mirror 105 and projected onto the front windshield 71. The concave mirror 105 has a function similar to that of a lens, and has a function of forming an image at a predetermined focal length. Therefore, assuming that the concave mirror 105 is a lens, the image on the screen 104 corresponding to the object is formed at the position of the _{distance R 2 determined by the focal length of the concave mirror 105.} Therefore, when viewed from the occupant such as the driver V of the vehicle 8, the virtual image I is displayed at the position of the _{distance R 1} + R _{2 from the front windshield 71.} Assuming that the distance from the driver V of the vehicle 8 to the front windshield 71 is R ₃ , in FIG. 2, the virtual image I is located at the _{distance R (= R 1} + R ₂ + R ₃ ) from the viewpoint E of the driver V of the vehicle 8. Displayed (imaging).

At least a part of the light flux to the front windshield 71 is reflected toward the viewpoint E of the driver V of the vehicle 8. As a result, the driver V of the vehicle 8 can visually recognize the virtual image I in which the intermediate image of the screen 104 is enlarged through the front windshield 71. In other words, when viewed from the driver V of the vehicle 8, a virtual image I in which the intermediate image is enlarged and displayed is displayed through the front windshield 71.

In general, the front windshield 71 is not flat but slightly curved. Therefore, the image formation position of the virtual image I is determined not only by the focal length of the concave mirror 105 but also by the curved surface of the front windshield 71, but the distance R is substantially determined by the _{distance R 1} + R _{2 as described above.} In order to image the virtual image I in the distance so that the movement of the line of sight is small, the distance R ₁ or R ₂ is lengthened. Distance R ₁ as a method of long there is a way to fold the optical path by a mirror, the distance R2 as a method for long, there is a method of adjusting the focal distance of the concave mirror 105.

Since the front windshield 71 causes optical distortion in which the horizontal line of the intermediate image becomes convex upward or downward, at least one of the mirror 103 and the concave mirror 105 is designed and arranged so as to correct the distortion. Is preferable. Alternatively, it is preferable that the projected image is corrected in consideration of distortion.

Further, a combiner may be arranged as a transmission reflection member on the viewpoint E side of the front windshield 71. Even if the combiner is irradiated with the light from the concave mirror 105, the information can be presented as a virtual image I as in the case where the front windshield 71 is irradiated with the light from the concave mirror 105.

<Configuration example of in-vehicle system 2 on which the display device 1 is mounted>
Next, an in-vehicle system 2 in which the display device 1 is mounted on the vehicle 8 will be described with reference to FIG. FIG. 3 is a block diagram showing an example of the configuration of the in-vehicle system 2 in which the display device 1 is mounted on the vehicle 8.

The in-vehicle system 2 includes a car navigation system 11 that communicates via an in-vehicle network NW such as a CAN (Controller Area Network) bus, an engine ECU (Electronic Control Unit) 12, a display device 1, a brake ECU 13, and a steering ECU 14. It includes an image processing device 15 and a detection device 16.

The car navigation system 11 includes a GNSS (Global Navigation Satellite System) represented by GPS (Global Positioning System), detects the current location of the vehicle 8, and displays the position of the vehicle 8 on an electronic map.

In addition, it accepts input of the departure point and the destination, searches for the route from the departure point to the destination, displays the route on the electronic map, and gives the occupant of the vehicle 8 the direction of travel by voice and text before changing the course. (Shown on the display) or guide by animation. The car navigation system 11 may communicate with the server via a mobile phone network or the like. In this case, the server can send an electronic map to the vehicle 8 or perform a route search.

The engine ECU 12 determines an ideal fuel injection amount, controls the advance / retard of the ignition timing, the valve operating mechanism, and the like according to the information from each sensor and the situation of the vehicle 8. In addition, the necessity of shifting is determined by referring to a map in which the switching line of the shifting stage is defined for the relationship between the current vehicle speed and the accelerator opening. The engine ECU 12 combines these to perform acceleration and deceleration control during follow-up traveling to the preceding vehicle. An electric motor may be used as a power source together with the engine or without using the engine as a power source.

The brake ECU 13 includes ABS (Antilock Braking System) control, braking control when following a preceding vehicle, automatic braking based on TTC (Time To Collision) with an obstacle, and stop state maintenance control when starting on a slope. The braking force is controlled for each wheel of the vehicle 8 without the operation of the brake pedal by the driver V of the vehicle 8.

The steering ECU 14 detects the steering direction and steering amount of the steering wheel by the driver V of the vehicle 8 and performs power steering control to add steering torque to the steering direction. Further, the vehicle 8 is steered in the direction of avoiding deviation from the traveling lane, in the direction of maintaining the traveling in the center of the traveling lane, or in the direction of avoiding approaching an obstacle, even if the driver V of the vehicle 8 does not operate the steering wheel. ..

The detection device 16 is provided with various sensors such as a front camera 164, and can perform various detections such as capturing a front image of the vehicle 8. Here, the front camera 164 is attached to the vehicle 8 after being adjusted in position and inclination so that the front image corresponds to the foreground seen by the driver V through the front windshield 71.

The image processing device 15 executes image processing on the front image of the vehicle 8 captured by the front camera 164, acquires information used in generating the projected image, and outputs the information to the display device 1.

<Configuration example of detection device 16>
Next, the configuration of the detection device 16 will be described with reference to FIG. FIG. 4 is a block diagram showing an example of the configuration of the detection device 16. The detection device 16 relates to a vehicle speed sensor 161 that detects the vehicle speed displayed by the display device 1, a vehicle information sensor 162 that acquires vehicle information displayed by the display device 1, a radar sensor 163 and a front camera 164 that detect obstacles, and an occupant. An occupant status information sensor 165 that acquires occupant status information, which is information, a VICS receiver 166 (registered trademark. VICS: Vehicle Information and Communication System Center) that receives congestion information from the outside, and an external communication device that connects to the Internet, etc. It is equipped with 167 and the like.

It should be noted that the sensors included in the detection device 16 do not have to be collectively present in the detection device 16, and may be mounted on a vehicle.

The vehicle speed sensor 161 detects a magnet that rotates with the rotation of the shaft of the drive train system by a sensor unit fixed to the vehicle body, and generates a pulse wave proportional to the rotation speed. The vehicle speed can be detected by the number of pulses per unit time.

The vehicle information sensor 162 includes one or more sensors that detect vehicle information other than the vehicle speed sensor 161. For example, a fuel gauge sensor, a shift lever position sensor, an odometer, a trip meter, a winker sensor, a water temperature sensor, and the like. It suffices if each vehicle information can be acquired in these with a general configuration. The fuel gauge sensor detects the current residual fuel.

The shift lever position sensor detects the shift lever position operated by the driver V of the vehicle 8. The odometer accumulates the mileage of the vehicle 8 to provide the total mileage. The trip meter provides the section mileage from the initialization by the driver V of the vehicle 8 to the present.

The winker sensor detects the direction of the winker operated by the driver V of the vehicle 8. The water temperature sensor detects the temperature of the engine cooling water. These are only examples of information that can be acquired from the vehicle, and in addition, the information that can be acquired from the vehicle 8 can be vehicle information. For example, in an electric vehicle or a hybrid vehicle, the remaining battery level, the amount of regenerative power, the amount of power consumption, and the like can be obtained.

The front camera 164 is an imaging device that images the front of the vehicle 8 (the direction corresponding to the positive Y direction in FIG. 1). The front camera 164 is mounted, for example, on or near the back of the rearview mirror.

The front camera 164 may be a monocular camera or a stereo camera. In the case of a monocular camera or a stereo camera capable of acquiring distance information, the radar sensor 163 may not be provided. However, in addition to the front camera 164 that can acquire the distance information, the radar sensor 163 fuses (integrates) the distance information of the front camera 164 and the distance information of the radar sensor 163 to complement each other's disadvantages. Highly accurate distance information can be acquired. In addition to the radar sensor 163 and the front camera 164, a sonic sensor (ultrasonic sensor) or the like may be provided.

The radar sensor 163 transmits radar to the surroundings of the vehicle 8, such as the front, side, and rear of the vehicle 8, and receives the radar reflected by an object and returned. The placement location may be any location where obstacles around the vehicle 8 can be detected. The radar sensor 163 has a TOF method (Time Of Fright) in which the distance to an object is detected by the time from transmission to reception and the direction of the object is detected by the irradiation direction of the radar.

LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) is known as a TOF type radar sensor. In addition, the FMCW method (Frequency Modulation Continuous Wave) that generates a mixed wave of the received wave and the transmitted wave while continuously increasing the frequency of the transmitted wave and converts the humming frequency of the mixed wave generated by the slightly different frequencies into a distance. ). In the FMCW method, the direction of an object is estimated by detecting the phase shift of the received wave with a plurality of receiving antennas.

The occupant state information sensor 165 is a sensor that detects occupant state information directly or indirectly detected from the occupant of the vehicle 8. A typical one is a face camera. The face camera captures the face of the occupant of the vehicle 8 and performs face recognition to identify or identify the occupant of the vehicle 8. In addition, it becomes possible to detect the face direction and the line-of-sight direction from the face image.

In addition, the occupant state information sensor 165 may include, for example, an electrocardiographic sensor, a heart rate sensor, a blood pressure sensor, a body temperature sensor, a pulse sensor, a breathing sensor, a sweating sensor, a blinking sensor, a pupil sensor, a brain wave sensor, a myoelectric potential sensor, or the like. It may be. As the occupant state information sensor 165, for example, there is a form in which a vehicle occupant wears a wristwatch-type wearable terminal (smart watch).

The VICS receiver 166 receives radio waves distributed by VICS. VICS is a system that transmits road traffic information such as traffic congestion and traffic regulation to an in-vehicle device in real time using FM multiplex broadcasting or a beacon. The external communication device 167 connects to the Internet or the like via a network such as 3G, 4G, 5G, LTE and wireless LAN, and receives various information. For example, it can receive weather information such as rain, snow, and fog.

You may also receive news, music, videos and the like. Further, the external communication device 167 can acquire, for example, the state information of the traffic light and the time until the signal changes. In this way, the VICS receiving device 166 and the external communication device 167 may perform road-to-vehicle communication. In addition, the external communication device 167 may acquire the information detected by the other vehicle 6 in the inter-vehicle communication.

In addition, the advanced driver assistance system (ADAS) not only displays information and warns, but also controls the vehicle 8. In this case, the ECU cooperates with the engine ECU 12, the brake ECU 13, and the steering ECU 14 to provide various driving support based on the distance information of the obstacle detected by at least one of the radar sensor 163 and the front camera 164. For example, acceleration / deceleration control, automatic braking, avoidance of deviation from the traveling lane, lane keeping traveling, and obstacle avoidance steering during follow-up traveling to the preceding vehicle are performed. For such control, the ECU recognizes road paint such as white lines from the image captured by the front camera 164.

In acceleration / deceleration control during follow-up travel, the ECU controls power and braking force so as to maintain a target distance according to the vehicle speed. In automatic braking, a warning, a display prompting the brake pedal to be pressed, a seatbelt winding and collision avoidance braking when there is a high possibility of a collision, and the like are performed according to the TTC. In avoiding deviation from the traveling lane, the driving support ECU 36 recognizes a white line (traveling division line) from the captured image and applies steering torque in a direction opposite to the direction deviating from the traveling lane.

Further, in lane-keeping driving, the center of the traveling lane is set as the target traveling line, and steering torque proportional to the deviation from the target traveling line is added in the direction opposite to the deviation. In obstacle avoidance steering, when it is determined that a collision cannot be avoided by braking, a traveling line for avoiding obstacles is determined, and steering torque traveling on this traveling line is added.

Further, for example, when the radar sensor 163 or the front camera 164 detects a vehicle traveling in parallel in an area (blind spot area) that is not reflected in the door mirror of the adjacent lane when changing lanes, the occupant is warned. Such support is called a blind spot monitor.

<Example of functional configuration of image processing device 15>
Next, the functional configuration of the image processing device 15 will be described. The image processing device 15 executes a process of calculating and acquiring position information of a road and a traffic sign in front of the vehicle 8 and outputting the information to the display device 1.

FIG. 5 is a block diagram showing an example of the functional configuration of the image processing device 15. As shown in FIG. 5, the image processing device 15 includes a sign area extraction unit 151, a road area extraction unit 152, a sign position acquisition unit 153, and a road position acquisition unit 154.

The sign area extraction unit 151 extracts an image area including a traffic sign in the front image of the vehicle 8 as a sign image, and outputs the sign image to the sign position acquisition unit 153. Further, the road area extraction unit 152 extracts an image area including the road in the front image of the vehicle 8 as a road image, and outputs the road image to the road position acquisition unit 154. Image processing such as pattern matching and edge detection can be applied to the extraction of the sign image and the road image. When a plurality of roads and a plurality of signs are included in the forward image, the sign image and the road image are extracted for each of the plurality of signs and the plurality of roads. The sign image includes coordinate information representing the position of the sign image in the front image, and the road image includes coordinate information representing the position of the road image in the front image.

Further, as one of the methods for the sign area extraction unit 151 and the road area extraction unit 152 to extract the image area, an image recognition method using machine learning can also be applied. Machine learning is a technology for making a computer acquire learning ability like a human being, and the computer autonomously generates an algorithm necessary for judgment such as data identification from the learning data taken in advance, and newly It is a technology that applies this to forecasts of various types of data.

The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, intensive learning, and deep learning, and may be a learning method that combines these learning methods. Machine learning It doesn't matter how you learn for. Machine learning methods include perceptrons, deep learning, support vector machines, logistic regression, naive bays, decision trees, and random forests.

The sign position acquisition unit 153 acquires the position information of the traffic sign in the front image of the vehicle 8 as coordinate information based on the sign image. The road position acquisition unit 154 acquires the position information of the road in the front image of the vehicle 8 as coordinate information based on the road image. When a plurality of roads and a plurality of signs are included in the forward image, position information is acquired for each of the plurality of roads and the plurality of signs.

The image processing device 15 outputs the sign image extracted by the sign area extraction unit 151, the position information of the traffic sign acquired by the sign position acquisition unit 153, and the road position information acquired by the road position acquisition unit 154 to the display device 1. do.

When the traffic sign is not included in the front image, the image processing device 15 outputs no sign information such as coordinates (0, 0) as information on the sign position. Further, when the road is no longer included in the front image, no road information may be output.

<Hardware configuration example of control unit 20>
Next, the hardware configuration of the control unit 20 will be described with reference to FIG. FIG. 6 is a diagram showing an example of the hardware configuration of the control unit 20. As shown in FIG. 6, the control unit 20 includes an FPGA (Field-Programmable Gate Array) 201, a CPU (Central Processing Unit) 202, a ROM (Read Only Memory) 203, a RAM (Random Access Memory) 204, and the like. It includes an I / F (Interface) 205, a bus line 206, an LD driver 207, a MEMS controller 208, and an HDD (Hard Disk Drive) 210. The FPGA 201, CPU 202, ROM 203, RAM 204, I / F 205 and HDD 210 are electrically connected to each other via the bus line 206.

The CPU 202 controls each function of the control unit 20. The ROM 203 stores a program 203p executed by the CPU 202 to control each function of the control unit 20. The program 203p is expanded in the RAM 204, and is used as a work area for the CPU 202 to execute the program 203p. Further, the RAM 204 has an image memory 209.

The image memory 209 is used to generate an image projected as a virtual image I. The I / F 205 is an interface for communicating with the image processing device 15 and the detection device 16, and is connected to, for example, the CAN (Controller Area Network) bus of the vehicle 8 or Ethernet (registered trademark).

The HDD 210 is a non-volatile memory that stores various data such as a program that executes processing by the control unit 20 and image data of a traffic sign, and is a large-capacity storage device. The HDD may be an SSD (Solid State Drive) or the like.

The FPGA 201 controls the LD driver 207 based on the image generated by the CPU 202. The LD driver 207 controls the light emission of the LD according to the image by driving the LD of the light source unit 101 of the projection unit 10. The FPGA 201 operates the light deflector 102 of the projection unit 10 via the MEMS controller 208 so that the laser beam is deflected in the direction corresponding to the pixel position of the image.

[First Embodiment]
<Example of functional configuration of control unit 20>
Next, the functional configuration of the control unit 20 according to the first embodiment will be described with reference to FIG. 7. FIG. 7 is a block diagram showing an example of the functional configuration of the control unit 20. As shown in FIG. 7, the control unit 20 includes an image storage unit 21, an identification unit 22, a direction target determination unit 23, a direction table storage unit 24, a direction instruction unit 25, an image generation unit 26, and an output unit. It has 27 and.

Of these, the functions of the identification unit 22, the direction target determination unit 23, the direction instruction unit 25, and the image generation unit 26 are realized by the CPU 202 executing a predetermined program or the like. Further, the functions of the image storage unit 21 and the direction table storage unit 24 are realized by the HDD 210 or the like, and the functions of the output unit 27 are realized by the I / F 205 or the like.

The image storage unit 21 is a database that stores virtual sign images corresponding to traffic signs. In addition, identification information for identifying the traffic sign is given to each of the stored traffic sign images. The virtual sign image may be a small size image within a range that can be visually recognized by the driver V when the virtual image of the projected image is visually recognized by the driver V, which is arranged in the projected image.

The image storage unit 21 preferably stores images of all traffic signs used in road traffic at that time, but is not limited to this, and is not limited to this, and is frequently used. Some may be stored. In addition, it is preferable that the stored image is updated regularly in order to correspond to the change of the traffic sign and the addition of a new traffic sign.

This update work is performed by downloading the latest data from the Internet or the like via the external communication device 167. The update may be performed automatically on a regular basis, or the driver V of the vehicle 8 may operate the in-vehicle system 2 to perform the update work manually.

Based on the sign image input from the image processing device 15, the identification unit 22 refers to the image storage unit 21 and acquires a virtual sign image and identification information corresponding to the traffic sign indicated by the sign image. As an example, the input sign image and the image of each traffic sign stored in advance in the image storage unit 21 are pattern-matched, and an image whose similarity is equal to or higher than a predetermined threshold is virtually labeled. Get as an image. In addition, the identification information given to the acquired virtual sign image can be acquired.

The identification unit 22 outputs the acquired virtual sign image to the image generation unit 26, and outputs the identification information to the direction target determination unit 23.

Based on the traffic sign identification information input from the identification unit 22, the direction target determination unit 23 determines either the direction in which the target road exists or the direction in which the traffic sign exists as the existence direction indicated to the driver V. decide. In other words, either the target road or the traffic sign is determined as the target of the direction of existence. Then, the information indicating the target of the determined existence direction is output to the direction indicating unit 25.

Specifically, the direction target determination unit 23 is a direction table storage unit that stores a correspondence table showing the correspondence relationship between the predetermined identification information and the existence direction based on the identification information of the traffic sign input from the identification unit 22. 24 makes the above determination.

For example, in the case of a traffic sign applied to a road such as no entry or temporary stop, the target road is the target in the direction of existence. On the other hand, in the case of a traffic sign that calls attention according to the surrounding environment of the road such as a school nearby or a pedestrian crossing nearby and assists traffic rules, the traffic sign is the target of the direction of existence. As for the speed limit traffic sign or the like, either the target road or the traffic sign may be the target in the existing direction, and the manufacturer of the display device 1, the manufacturer of the vehicle 8, or the display device 1 determines which one to use. It is preferable to set in advance by the setting of the user or the like.

The direction indicating unit 25 drives the target road to which the traffic sign is applied or the existing direction in which either the traffic sign is present, based on the position of the traffic sign and the position of the road in front of the vehicle 8. It has the function shown in Person V.

Specifically, the direction indicating unit 25 inputs the position information of each of the traffic sign and the road from the image processing device 15, and the direction target determining unit 23 inputs the target information of the existing direction.

When the target in the existing direction is the target road, the direction indicating unit 25 determines the road to which the traffic sign is applied as the target road based on the position of the traffic sign and the position of the road in front of the vehicle 8. .. Then, using the predetermined reference position and the coordinate information of the target road, the direction in which the target road exists with respect to the reference position is calculated and acquired as the existence direction θ.

On the other hand, when the target of the existence direction is a traffic sign, the direction indicating unit 25 calculates and acquires the direction of the traffic sign with respect to the reference position as the existence direction θ by using the reference position and the coordinate information of the traffic sign. ..

Here, the viewpoint (eye) of the driver V or the position corresponding to the viewpoint can be used as the reference position. In this case, the direction in which the target road or the traffic sign is present corresponds to the existing direction when viewed from the driver V.

The direction indicating unit 25 outputs the existence direction θ to the image generation unit 26, and causes the image generation unit 26 to generate a projected image including the existence direction θ, thereby indicating the existence direction θ to the driver V.

The image generation unit 26 has a function of generating a projected image in which a virtual sign image is arranged in the image. Specifically, the image generation unit 26 is a projected image having a predetermined image size in which the reference position is aligned with the front image, and the virtual marker image is arranged in the direction of the existence direction θ from the reference position. Generate an image. The predetermined image size is preferably an image size in which the size of the virtual image of the projected image is smaller than that of the foreground in order to narrow the projection angle of view to the front windshield 71 by the projection unit 10.

The placement of the virtual sign image in the projected image is executed by replacing the image area in the direction of the existence direction θ from the reference position with the virtual sign image. The image generation unit 26 can also arrange other information such as a vehicle speed display in the projected image. Further, the image generation unit 26 generates a projected image that does not include the virtual sign image when the traffic sign is not included in the front image.

The output unit 27 outputs the image data of the projected image generated by the image generation unit 26 to the projection unit 10. That is, the LD driver 207 and the MEMS controller 208 are controlled to project the projected image generated by the image generation unit 26 onto the projection unit 10 and display it as a virtual image.

<Processing example by control unit 20>
Next, the flow of processing by the control unit 20 will be described with reference to FIG. FIG. 8 is a flowchart showing an example of processing by the control unit 20.

First, in step S81, the identification unit 22 inputs a marker image from the image processing device 15.

Subsequently, in step S82, the identification unit 22 refers to the image storage unit 21 based on the sign image and acquires a virtual sign image and identification information corresponding to the traffic sign indicated by the sign image. Then, the acquired virtual sign image is output to the image generation unit 26, and the identification information is output to the direction target determination unit 23.

Subsequently, in step S83, the direction target determination unit 23 determines either the target road or the traffic sign as the target of the existence direction indicated to the driver V based on the identification information of the traffic sign input from the identification unit 22. .. Then, the information indicating the target of the determined existence direction is output to the direction indicating unit 25.

Subsequently, in step S84, the direction indicating unit 25 inputs the position information of the traffic sign and the road from the image processing device 15. Note that this step S84 may be performed at any timing between steps S81 to S83 described above.

Subsequently, in step S85, the existence direction in which either the target road to which the traffic sign is applied or the traffic sign exists is acquired. When the target in the existing direction is the target road, the direction indicating unit 25 determines the road to which the traffic sign is applied as the target road based on the position of the traffic sign and the position of the road in front of the vehicle 8. .. Then, using the predetermined reference position and the coordinate information of the target road, the direction in which the target road exists with respect to the reference position is calculated and acquired as the existence direction θ.

On the other hand, when the target of the existence direction is a traffic sign, the direction indicating unit 25 calculates and acquires the direction of the traffic sign with respect to the reference position as the existence direction θ by using the reference position and the coordinate information of the traffic sign. ..

After that, the direction indicating unit 25 outputs the existence direction θ to the image generation unit 26.

Subsequently, in step S86, the image generation unit 26 generates a projected image in which the virtual marker image is arranged in the direction of the existence direction θ from the reference position.

Subsequently, in step S87, the output unit 27 outputs the image data of the projected image generated by the image generation unit 26 to the projection unit 10.

In this way, the display device 1 can generate a projected image for showing the driver the direction in which either the target road to which the traffic sign is applied or the traffic sign exists. Then, a virtual image of the projected image by the projection unit 10 can be displayed.

The projected image is updated at any time according to the front image that changes due to the movement of the vehicle 8, and the virtual image of the updated projected image is superimposed and displayed on the foreground. When the traffic sign is no longer included in the front image, the virtual image of the projected image that does not include the virtual sign image is superimposed and displayed on the foreground.

<Application example of the first embodiment>
Next, an application example of the display device 1 will be described with reference to specific examples of the front image, the projected image, and the foreground in which the projected image is superimposed and displayed.

FIG. 9 is a diagram showing an example of a front image 80 captured by the front camera 164. The front image 80 includes a preceding vehicle 81, traffic signs 82 and 83, and roads 84 and 85. The traffic sign 82 is a road sign applied to the road 84, and the traffic sign 83 is a road sign applied to the road 85.

The sign area extraction unit 151 in the image processing device 15 extracts the sign image 82a including the traffic sign 82 and the sign image 83a including the traffic sign 83 in the front image 80, and puts each of them in the sign position acquisition unit 153. Output. Further, the road area extraction unit 152 extracts the road image 84a including the road 84 and the road image 85a including the road 85 in the front image 80, and outputs each of them to the road position acquisition unit 154.

The sign position acquisition unit 153 acquires the coordinate information of the sign position 82b, which is the position of the traffic sign 82 in the sign image 82a. Further, the coordinate information of the sign position 83b, which is the position of the traffic sign 83 in the sign image 83a, is acquired. Such a sign position is, for example, the position of the pixel on the most positive X direction side and the most negative Y direction side among the pixels constituting the traffic sign in the sign image. Which pixel position is used as the marker position is predetermined.

Similarly, the road position acquisition unit 154 acquires the coordinate information of the road position 84b, which is the position of the road 84 in the road image 84a. Further, the coordinate information of the road position 85b, which is the position of the road 85 in the road image 85a, is acquired. Such a road position is, for example, a position corresponding to the center of gravity of the road image. Which pixel position is set as the road position is predetermined.

The identification unit 22 appropriately corrects the labeled image 82a and the labeled image 83a by preprocessing such as image rotation so that the pattern matching process can be easily executed. After that, based on the sign images 82a and 83a, the virtual sign image and the identification information are acquired with reference to the image storage unit 21.

In the example of FIG. 9, since the traffic signs 82 and 83 are traffic signs indicating one-way traffic, the identification unit 22 is a one-way virtual sign image as a virtual sign image corresponding to each of the traffic signs 82 and 83. To get. Alternatively, as a virtual sign image corresponding to one-way traffic, a virtual sign image indicating that the road cannot be entered due to one-way traffic may be acquired. The acquisition rule of such a virtual sign image is predetermined.

Since the one-way traffic sign is a traffic rule applied to the road, the direction target determination unit 23 uses the traffic signs 82 and 83 based on the identification information of the traffic signs 82 and 83 input from the identification unit 22. The target road is determined as the target of each direction of existence.

In the front image 80, the direction indicating unit 25 determines the road 84 closest to the sign position 82b as the target road for the traffic sign 82, and the road 84 closest to the sign position 83b is the target for the traffic sign 83. Determined as a road. In the example of FIG. 9, the road 84 and the road 85 are at the same distance from the sign position 82b, but the road 84 is determined as the target road in consideration of the overlap with the target road of the traffic sign 83.

Reference position V _C in FIG. 9 is a position corresponding to the viewpoint of the driver V (eye). The coordinates of the road location _{84b (X 1, Y 1)} , the coordinates of the road position _{85b (X 2, Y 2)} , the reference position _{V C} of coordinates _(X C, _{Y C)} When the direction instruction section 25 road 84 The existence direction θ ₁ of the road 85 and the existence direction θ ₂ of the road 85 are calculated using the following equations (1) and (2), respectively.
θ ₁ = arctan {(Y _1- Y _C ) / (X _1- X _C )} ... (1)
θ ₂ = arctan {(Y _2- Y _C ) / (X _2- X _C )} ... (2)
FIG. 10 is a diagram showing an example of the projected image 90 generated by the image generation unit 26. _{The projected image 90 includes a virtual sign image 91 arranged in the existence direction θ 1} of the road 84 to which the traffic sign 82 is applied, a graphic image 92, and a vehicle speed image 93 arranged at a predetermined position. It is composed of. Here, the virtual sign image 91 is an example of a predetermined image.

The virtual sign image 91 virtually displays a no-entry road sign indicating that the person cannot enter because it is a one-way street. The graphic image 92 is a graphic image showing the _{existence direction θ 1.} The vehicle speed image 93 is a character image showing the vehicle speed of the vehicle 8 detected by the vehicle speed sensor 161. The vehicle speed image 93 is fixed at a predetermined position, and the position does not change even if the vehicle 8 moves and the front image changes. Such a vehicle speed image 93 is an example of a fixed image.

In the present embodiment, the graphic image 92 illustrates a method of displaying an arrow and a three-dimensional conical figure to indicate a direction, but any method other than these may be used as long as the direction can be indicated. .. For example, a language such as "right" or "left front" can be displayed to display a character image indicating a direction. In this case, for example, the display device 1 replaces the arrow and the three-dimensional conical figure, or together with the arrow and the three-dimensional conical figure, "right front", "Right front", or "horizontal to right". It is possible to display a character image which is a character or a language such as "direction of an angle θ _{1 degree".} The character image 94 displaying the character “right front” in FIG. 10 is an example of such a character image.

Next, FIG. 11 is a diagram showing an example of the foreground 100 of the vehicle 8 in the present embodiment. The foreground 100 is an image in which the display device 1 projects the projected image 90 onto the front windshield 71 so that the virtual image 90'of the projected image 90 is superimposed and displayed and visually recognized by the driver V. As shown in FIG. 11, the size of the virtual image 90'is smaller than that of the foreground 100.

A virtual sign virtual image 91'which is a virtual image of the virtual sign image 91 and a virtual image 92' which is a virtual image of the graphic image 92 are arranged in a direction along _{the existence direction θ 1} of the road 84 which is the target road to which the traffic sign 82 is applied. It is displayed. Further, a vehicle speed virtual image 93'which is a virtual image of the vehicle speed image 93 is displayed at a predetermined position.

By displaying the virtual sign virtual image 91'corresponding to the traffic sign 82 in the direction _{along the existence direction θ 1} , the driver V can recognize the traffic sign 82 without moving his / her line of sight, and the traffic sign 82 is applied. The road 84 to be used can be recognized.

However, the virtual sign virtual image 91'does _{not have to be displayed exactly in the direction along the existence direction θ 1} , and if the driver V can recognize the direction in which the target road or the traffic sign exists, the virtual sign is slightly deviated from _{the existence direction θ 1.} It may be displayed in the same direction.

Here, for example, when an attempt is made to display a virtual sign virtual image 91'or a graphic virtual image 92'by superimposing it on a road 84 in the foreground 100, a virtual image larger than the virtual image 90'is obtained, and the projection angle of view by the projection unit 10 is obtained. Is required to have a wider angle.

On the other hand, in the embodiment, _{by indicating the existing direction θ 1} of the road 84, the virtual sign virtual image 91'or the graphic virtual image 92'is not superimposed and displayed at the position where the road 84 is visually recognized in the foreground 100. , The driver V is made to recognize the road 84 to which the traffic sign 82 is applied.

FIG. 12 is a diagram showing an example of the foreground 100 after the vehicle 8 has moved from the state of FIG. Foreground 100 is changed according to the movement of the vehicle 8 has changed from the presence direction theta ₁ of the road 84 as viewed from the driver V to theta _11. The position where the virtual sign virtual image 91'is displayed changes according to the change in the existence direction.

Further, the position of the virtual image 92'changes according to the change in the direction of existence, and the shape also changes. By changing the three-dimensional conical shape of the graphic virtual image 92', the reality can be further improved and the direction can be indicated as compared with the case where the direction is indicated by a simple arrow or the like. The position of the vehicle speed virtual image 93'is fixed regardless of the movement of the vehicle 8, and only the numerical value indicating the vehicle speed changes.

The virtual sign virtual image 91'and the graphic virtual image 92'are arranged around the vehicle speed virtual image 93'on the line connecting the reference position Vc and the road position 84b, and on the circumference around the vehicle speed virtual image 93' depending on the direction of existence. The position changes.

_{Although FIGS. 9 and 10 show an example showing only the existence direction θ 1 of the} road 84 to which the traffic sign 82 is applied _{, the existence direction θ 2 of the} road 85 to which the traffic sign 83 is applied is also shown in the same manner. be able to.

<Operation and effect of display device 1>
Next, the operation and effect of the display device 1 will be described.

Display devices such as head-up displays recognize traffic signs such as road signs and road markings in front of the vehicle based on the vehicle front image acquired by the camera attached to the vehicle, and position the road so that it can be seen in the foreground. Some virtual sign images are superimposed and displayed.

However, in such a device, when the road to which the traffic sign is applied or the traffic sign is visually recognized near the edge of the foreground, a virtual image of the virtual sign image is superimposed and displayed on the position of the traffic sign in the foreground. Is required. For example, it is required to superimpose and display a virtual image having a size larger than the virtual image 90'of the projected image in FIG.

Here, FIG. 13 is a diagram for explaining the widening of the projection unit 10. As shown in FIG. 13, if the driver V wants to visually recognize a virtual image I'that is larger in size than the virtual image I, a projection angle of view φ'that is wider than the projection angle of view φ for the virtual image I is required. If the projection angle of view is widened, the display device may become large, or a complicated and expensive projection lens may be required to ensure the quality of the virtual image.

On the other hand, in the present embodiment, the driver V is shown the direction in which either the target road or the traffic sign exists. For example, a virtual marker image is arranged in the image along the existence direction, and a projected image in which the figure indicating the existence direction is included in the image is generated by the image generation unit 26, and a virtual image of the projected image is displayed. By doing so, the direction of existence is indicated.

By indicating the direction, it is not necessary to superimpose a virtual image of a predetermined image such as a virtual sign image on the position of the target road visually recognized in the foreground, and it is not necessary to widen the projected angle of view. As a result, even if the projection angle of view is narrow, such as the projection angle of view φ in FIG. 13, either the target road or the traffic sign can be indicated.

On the other hand, traffic signs are applied to roads such as prohibition of entry and temporary stop, and auxiliary ones that call attention according to the surrounding environment of the road such as having a school nearby and a pedestrian crossing nearby. There is. In the case of a traffic sign applied to a road, it is better to be able to recognize the target road, but in the case of an auxiliary traffic sign, it is better to be able to recognize the traffic sign itself rather than the target road.

On the other hand, the present embodiment includes a direction target determination unit that determines either the target road or the traffic sign as the target of the existing direction based on the traffic sign. This makes it possible to indicate the direction of existence of the target road when applied to a road, and to indicate the direction of existence of a traffic sign in the case of an auxiliary traffic sign, and the driver can provide appropriate information according to the traffic sign. Can be provided to V.

Further, in the present embodiment, the existence direction that changes according to the movement of the vehicle 8 is shown. As a result, it is possible to provide the driver V with appropriate information according to the change in the situation due to the movement of the vehicle 8.

Further, in the present embodiment, the virtual sign image 91 is arranged along the existence direction around the fixed image such as the vehicle speed image 93 fixedly arranged in the projected image to indicate the existence direction. The fixed image is often visually recognized by the driver V, and is often fixed at a position that is easy to see. By arranging the virtual sign image 91 around such a fixed image, the driver V can easily see the direction of existence.

In the present embodiment, an example in which the virtual sign image 91 and the graphic image 92 are displayed together as a predetermined image is shown, but either one may be displayed. Further, the graphic image 92 may be a simple graphic image of an arrow or a character image indicating an existence direction such as an angle display. Further, although an example of changing the position of the virtual sign image 91 along the existence direction to indicate the existence direction is shown, the existence direction may be indicated only by rotating the virtual sign image 91 without changing the position.

Further, in the present embodiment, an example in which the image processing device 15 acquires the traffic sign and the position information of the road based on the image captured by the front camera 164 is shown, but the present invention is not limited to this. Using the information input from the car navigation system 11, the position information of the road and the traffic sign in front of the vehicle 8 may be acquired. Further, the vehicle 8 may be provided with a gyro sensor for detecting the posture of the vehicle, and the detected value of the gyro sensor may be used.

Further, although an example in which the image storage unit 21 stores the virtual sign image is shown, the present invention is not limited to this. A character image expressing the information indicated by the traffic sign in characters may be stored, or a graphic image for displaying the traffic sign in a simplified manner may be stored.

Further, an example is shown in which the identification unit 22 refers to the image storage unit 21 to acquire the virtual sign image and the identification information, and the direction target determination unit 23 refers to the direction table storage unit 24 to determine the target in the existing direction. However, it is not limited to this. The identification unit 22 and the direction target determination unit 23 may realize the functions by the method using machine learning described above.

Further, in the present embodiment, an example in which the existence direction θ is acquired by two-dimensional image processing using an image captured by the monocular front camera 164 is shown, but the present invention is not limited to this. Three-dimensional based on the projection relationship using the distance information to at least one of the target road or the traffic sign detected by the stereo camera as the front camera 164 and the known positional relationship between the vehicle 8 and the driver V. The existence direction θ may be acquired by image processing. By doing so, the virtual image of the predetermined image can be visually recognized by the driver V with higher reality, and the existence direction can be indicated.

[Second Embodiment]
Next, the display device 1a according to the second embodiment will be described.

When the vehicle 8 moves, the traffic sign is not included in the foreground of the vehicle 8, and the traffic sign information may not be acquired by the front camera 164. In this case, the virtual sign image corresponding to the traffic sign for which information is no longer acquired may be hidden, but in the case of a traffic sign applied to a certain section of the road such as the speed limit, the information is no longer acquired. After that, it may be preferable to make the virtual image of the virtual sign image visible to the driver V.

In the present embodiment, when the traffic sign information is no longer acquired by the front camera 164 due to the movement of the vehicle 8, the virtual sign image is generated as a projected image fixed at a predetermined position in the image. As a result, the driver V can visually recognize the virtual image of the virtual sign image even when the information is no longer acquired.

Further, depending on the traffic sign, either a projected image in which the virtual sign image is fixed at a predetermined position in the image is generated, or a projected image that does not include the virtual sign image is selected. do. As a result, when the information is no longer acquired, either the virtual image of the virtual sign image or the hidden virtual sign image can be selected according to the traffic sign.

<Example of functional configuration of control unit 20>
FIG. 14 is a block diagram showing an example of the functional configuration of the control unit 20a included in the display device 1a. As shown in FIG. 14, the control unit 20a includes a continuation table storage unit 28, an information holding unit 29, a direction indicating unit 25a, and an image generation unit 26a. Further, the direction indicating unit 25a includes a determination unit 241 and a selection unit 242.

Of these, the function of the continuation table storage unit 28 is realized by the HDD 210 or the like, and the function of the information holding unit 29 is realized by the RAM 204 or the like.

The continuation table storage unit 28 stores a correspondence table showing the correspondence relationship between the predetermined identification information and the display continuation information. Here, the display continuation information refers to information indicating either the virtual sign image is fixed at a predetermined position or the virtual sign image is hidden. For example, when the virtual sign image is fixed, the display continuation information is set to 1, and when it is hidden, the display continuation information is set to 0.

The information holding unit 29 temporarily holds the identification information of the traffic sign before the movement of the vehicle 8. Since the vehicle 8 is constantly moving during traveling, if the identification information of the traffic sign before the movement of the vehicle 8 is the identification information one cycle before a predetermined time in the imaging cycle of the front camera 164, it is immediately before the movement. It is suitable because identification information can be obtained.

The direction indicating unit 25a inputs identification information from the direction target determining unit 23. Then, when the traffic sign information is acquired, the identification information is output to the information holding unit 29 and held. On the other hand, when the traffic sign information is no longer acquired, the display continuation information is output to the image generation unit 26a. In addition, the direction indicating unit 25a has the same function as the direction indicating unit 25.

The determination unit 241 in the direction indicating unit 25a determines whether or not the traffic sign information has been acquired based on the sign position information input from the image processing device 15, and outputs the determination result to the selection unit 242.

Specifically, when the traffic sign is not included in the foreground of the vehicle 8 and the traffic sign information is not acquired by the front camera 164, the image processing device 15 outputs the no sign information as described with reference to FIG. The determination unit 241 determines that the traffic sign information has not been acquired when the no-sign information is input.

When the traffic sign information is not acquired, the selection unit 242 selects either fixing the virtual sign image at a predetermined position or hiding the virtual sign image based on the traffic sign. , The selection result is output to the image generation unit 26.

Specifically, when it is determined that the traffic sign information has not been acquired, the selection unit 242 accesses the information holding unit 29 and acquires the identification information of the traffic sign temporarily held by the information holding unit 29. do. Then, based on this identification information, the display continuation information is acquired by referring to the continuation table storage unit 28. This display continuation information corresponds to the above selection result.

When the display continuation information (for example, 1) indicating that the virtual sign image is fixed at a predetermined position is input, the image generation unit 26a generates a projected image in which the virtual sign image is fixed at a predetermined position in the image. do. When display continuation information (for example, 0) indicating that the virtual sign image is to be hidden is input, a projected image that does not include the virtual sign image is generated. On the other hand, when the information of the traffic sign is acquired (when the information of the sign position is not the information without the sign), the projected image in which the virtual sign image is arranged in the direction of the existence direction θ from the reference position is generated.

<Processing example by control unit 20a>
Next, the flow of processing by the control unit 20a will be described with reference to FIG. FIG. 15 is a flowchart showing an example of processing by the control unit 20a. Since the processes in steps S151 to S154 in FIG. 15 are the same as the processes in steps S81 to S84 in FIG. 8, redundant description will be omitted here. In addition to the above steps, the description of the portion overlapping with the description in FIG. 8 will be omitted as appropriate.

In step S155, the determination unit 241 determines whether or not the traffic sign information has been acquired based on the sign position information input from the image processing device 15.

If it is determined in step S155 that the information has been acquired (step S155, Yes), in step S156, the direction indicating unit 25a outputs the identification information of the traffic sign to the information holding unit 29 and holds it.

Subsequently, in step S157, the direction indicating unit 25a acquires the existence direction θ in which either the target road or the traffic sign exists, and outputs this to the image generation unit 26a.

Subsequently, in step S158, the image generation unit 26a generates a projected image in which the virtual marker image is arranged in the direction of the existence direction θ from the reference position. After that, the process proceeds to step S163.

On the other hand, if it is determined in step S155 that the traffic sign information has not been acquired (step S155, No), in step S159, the selection unit 242 accesses the information holding unit 29 and the information holding unit 29 temporarily Obtain the identification information of the traffic sign you are holding.

Subsequently, in step S160, the selection unit 242 refers to the continuation table storage unit 28 based on the identification information to acquire the display continuation information.

Subsequently, in step S161, the image generation unit 26a determines whether or not to continuously display the traffic sign based on the display continuation information.

If it is determined in step S161 that continuous display is performed (step S161, Yes), in step S162, the image generation unit 26a generates a projected image in which the virtual marker image is fixed at a predetermined position in the image. On the other hand, if it is determined that the continuous display is not performed (step S161, No), in step S163, the image generation unit 26a generates a projected image that does not include the virtual sign image.

Subsequently, in step S164, the output unit 27 outputs the projected image generated by the image generation unit 26a to the projection unit 10.

In this way, in the display device 1a, when the traffic sign information is no longer acquired by the front camera 164 due to the movement of the vehicle 8, the virtual sign image is fixed at a predetermined position in the image according to the traffic sign. It is possible to generate either a projected image or a projected image that does not include a virtual sign image. Then, a virtual image of the projected image by the projection unit 10 can be displayed.

The virtual sign image fixed at a predetermined position can be hidden by the image generation unit 26a at the timing when the traffic sign information is newly acquired in the front image due to the movement of the vehicle 8.

<Application example of the second embodiment>
Next, an application example of the display device 1a will be described with reference to specific examples of the front image, the projected image, and the foreground in which the projected image is superimposed and displayed.

First, FIG. 16 is a diagram showing an example of the foreground 100 of the vehicle 8 in the present embodiment. The foreground 160 is an image in which a virtual image 171 of the projected image is superimposed and displayed by the display device 1a projecting an image on the front windshield 71, and is visually recognized by the driver V. In addition to the virtual image 171, the foreground 160 includes a traffic sign 172, a virtual sign virtual image 173, a graphic virtual image 174, and a vehicle speed virtual image 175.

In the example of FIG. 16, the traffic sign is defined as the target of the existence direction, and the virtual sign virtual image 173 and the graphic virtual image 174 are displayed along _{the existence direction θ 12 of the traffic sign 172.} Further, the vehicle speed virtual image 175 is fixedly displayed at a predetermined position.

Next, FIG. 17 is a diagram showing an example of the foreground 160 after the vehicle 8 has moved from the state of FIG. The foreground 160 changes according to the movement of the vehicle 8, and the traffic sign 172 in FIG. 16 is no longer included. That is, the traffic sign information is not acquired.

In the example shown in FIG. 17, since the traffic sign is a speed limit road sign and is applied to a certain section of the road, it can be continuously displayed even after the traffic sign information is no longer acquired. preferable. Therefore, it is defined in the corresponding table of the continuation table storage unit 28 so that the road sign of the speed limit is continuously displayed. Based on this corresponding table, the display device 1a fixes the virtual sign virtual image 183 next to the vehicle speed virtual image 182 and continuously displays it. Displaying the vehicle speed virtual image 182 next to it makes it easier for the driver V to compare the vehicle speed of the vehicle 8 with the speed limit, which is preferable.

Although FIG. 17 shows an example in which the virtual sign virtual image 183 is fixedly displayed next to the vehicle speed virtual image 182, the display position of the virtual sign virtual image 183 is not limited to this, and may be any position. .. A position that is easy for the driver V to see is preferable.

<Action and effect of display device 1a>
As described above, in the present embodiment, when the traffic sign information is no longer acquired due to the movement of the vehicle 8, the virtual sign image is generated as a projected image fixed at a predetermined position in the image. As a result, even when the traffic sign information is no longer acquired, the virtual image of the virtual sign image can be continuously visually recognized by the driver V.

Further, in the present embodiment, either the virtual sign image is fixed at a predetermined position or the virtual sign image is hidden according to the traffic sign. As a result, when the traffic sign information is no longer acquired, either the virtual image of the virtual sign image fixed at a predetermined position is displayed on the driver V, or the virtual sign image is hidden. Can be selected according to the traffic sign. Then, appropriate information can be provided to the driver V according to the traffic sign.

<Other preferred examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications and modifications and variations and within a range not deviating from the gist of the present invention are made. Substitutions can be added.

In the above-described embodiment, the control unit 20 may have a part or all of the functions provided by the image processing device 15. Further, a part of the functions provided by the control unit 20 may be provided by other components such as the image processing device 15, the engine ECU 12, and the detection device 16. Further, the information stored in the image storage unit 21, the direction table storage unit 24, and the like can be stored by the external server, and the stored information can be referred to by accessing the external server via the network.

Further, in the embodiment, the road is an example of a route, the vehicle is an example of a moving body, and the HUD is an example of a display device, but the present invention is not limited thereto. The embodiment can also be applied to routes other than roads, moving objects other than vehicles, and display devices other than HUDs.

The embodiment also includes a display method. For example, the display method is a display method using a display device that allows a moving body driver to visually recognize a virtual image of a projected image, and displays a predetermined image acquired based on a traffic sign that is at least one of a road sign or a road marking. Based on the step of generating the projected image including the projected image by the image generation unit, the position of the traffic sign, and the position of the road in front of the moving body, the target road to which the traffic sign is applied or the traffic sign A step of outputting the existence direction in which either one exists and causing the image generation unit to generate the projected image including the existence direction is performed. By such a display method, the same effect as that of the display device described above can be obtained.

The embodiment also includes a program. For example, in a program, an image generator generates a projected image including a predetermined image acquired based on a traffic sign which is at least one of a road sign or a road marking, and the position of the traffic sign and the front of a moving body are generated. Based on the position of a certain road, the existing direction in which either the target road to which the traffic sign is applied or the traffic sign exists is output, and the projected image including the existing direction is sent to the image generation unit. Have the computer execute the process to be generated. With such a program, the same effect as that of the display device described above can be obtained.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit modules.

1 Display device 2 In-vehicle system 8 Vehicle (an example of a moving body)
10 Projection unit 15 Image processing device 151 Label area extraction unit 152 Road area extraction unit 153 Marking position acquisition unit 154 Road position acquisition unit 16 Detection device 164 Front camera 20 Control unit 21 Image storage unit 22 Identification unit 23 Direction Target determination unit 24 Direction Table storage unit 25 Direction indication unit 26 Image generation unit 27 Output unit 28 Continuation table storage unit 29 Information retention unit 71 Front windshield 80 Front image 82 Traffic sign 84 Road (example of route)
90 Projected image 90'Virtual image 91 Virtual sign image (example of predetermined image)
91'Virtual sign virtual image 92 Graphic image 92' Graphic virtual image 93 Vehicle speed image 93'Vehicle speed virtual image 100 Foreground 210 HDD
Vc Reference position θ, θ ₁ , θ ₂ , θ ₁₁ , θ ₁₂ Existence direction φ Projection angle of view

Japanese Unexamined Patent Publication No. 2018-200626

Claims (14)

A display device that displays a virtual image of a projected image.
An image generation unit that generates the projected image including a predetermined image acquired based on a traffic sign, and an image generation unit.
Based on the position of the traffic sign and the position of the route in front of the moving body, the existence direction in which either the target route to which the traffic sign is applied or the traffic sign exists is output, and the existence direction is output. A display device including a direction indicating unit that causes the image generation unit to generate the projected image including the above. The display device according to claim 1, wherein the presence direction is a direction in which either the target route or the traffic sign is present when viewed from the driver of the moving body. The display device according to claim 1 or 2, wherein the predetermined image is a virtual sign image showing at least one of a figure or a character corresponding to the traffic sign. The display device according to any one of claims 1 to 3, further comprising an identification unit for identifying the traffic sign. The display device according to any one of claims 1 to 4, further comprising a direction target determination unit that determines either one of the target route or the traffic sign as a target in the existing direction based on the traffic sign. The display device according to any one of claims 1 to 5, wherein the direction indicating unit indicates the existing direction that changes according to the movement of the moving body. The direction indicating unit according to any one of claims 1 to 6, wherein the image generation unit generates the projected image including the graphic image or the character image indicating the existing direction. Display device. The direction indicating unit is any one of claims 1 to 7 indicating the existing direction by causing the image generation unit to generate the projected image in which the predetermined image is arranged in the image along the existing direction. The display device according to item 1. The direction indicating unit causes the projected image in which the predetermined image is arranged in the image along the existing direction to the image generation unit around the fixed image fixedly arranged in the projected image. The display device according to any one of claims 1 to 8, which indicates the direction of existence by generating the display device. The image generation unit
The display device according to claim 8 or 9, wherein when the information of the traffic sign is not acquired due to the movement of the moving body, the predetermined image is fixed at a predetermined position in the image to generate the projected image. .. The image generation unit
When the information of the traffic sign is not acquired due to the movement of the moving body, the projected image in which the predetermined image is fixed at a predetermined position in the image based on the traffic sign, or the predetermined image is included. The display device according to claim 8 or 9, which generates any one of the projected images. A mobile body having the display device according to any one of claims 1 to 11. It is a display method using a display device that displays a virtual image of the projected image.
A step of generating the projected image including a predetermined image acquired based on a traffic sign, and a step of generating the projected image.
Based on the position of the traffic sign and the position of the route in front of the moving body, the existence direction in which either the target route to which the traffic sign is applied or the traffic sign exists is output, and the existence direction is output. A display method for causing the image generation unit to generate the projected image including the above. The image generation unit generates a projected image including a predetermined image acquired based on the traffic sign, and the image generation unit generates the projected image.
Based on the position of the traffic sign and the position of the route in front of the moving body, the existence direction in which either the target route to which the traffic sign is applied or the traffic sign exists is output, and the existence direction is output. A program that causes a computer to execute a process of causing the image generation unit to generate the projected image including the above.

Images