JP2019168745A - Display device, mobile body, display method, and program - Google Patents

Abstract

To reduce an unnatural feeling of an object to be displayed by addition to a real environment.SOLUTION: A display device for displaying a predetermined object at a position to be superimposed on an external environment of a mobile body includes: an acquisition section for acquiring an external image of a mobile body; a determination section for determining an image in an area in the periphery of a predetermined object in accordance with an image acquired by the acquisition section; and a display section for displaying a predetermined object and an image of an area in the periphery of the predetermined object.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a display device, a moving body, a display method, and a program.

  2. Description of the Related Art Conventionally, in a moving body (moving device) such as a vehicle, a ship, an aircraft, and an industrial robot that carries a passenger such as a driver, a head-up display that displays an object for providing information to the passenger ( A device using HUD (Head-Up Display) is known. In this HUD, for example, display image light is reflected by a windshield or a combiner, and is visually recognized by a passenger of a moving body.

  In this HUD, in order to improve the visibility of an object to be displayed, a technique for adjusting the display brightness, display color, and the like of the object is known (see, for example, Patent Document 1).

  However, in the prior art, for example, when an object such as a route guidance figure is added and displayed in an actual environment such as in front of a moving object by AR (Augmented Reality), the object is displayed in the background. There is a problem that it may be difficult to perceive as it exists in the environment.

  Therefore, an object is to reduce the uncomfortable feeling of an object displayed in addition to an actual environment.

  A display device that displays a predetermined object at a position overlapping with an environment outside the moving body, an acquisition unit that acquires an image outside the moving body, and an image of an area around the predetermined object. And a display unit that displays the predetermined object and an image of an area around the predetermined object determined by the determination unit.

  According to the disclosed technology, it is possible to reduce a sense of incongruity of an object displayed in addition to an actual environment.

It is a figure which shows an example of the system configuration | structure of the display system which concerns on embodiment. It is a figure which shows an example of the range where an image is projected by the display apparatus which concerns on embodiment. It is a figure which shows an example of the hardware constitutions of the display apparatus which concerns on embodiment. It is a figure which shows an example of the functional block of the display apparatus which concerns on embodiment. It is a flowchart which shows an example of a process of the display apparatus which concerns on embodiment. It is a flowchart which shows an example of the process which produces | generates the blend area | region which concerns on embodiment. It is a figure explaining the process which produces | generates the blend area | region which concerns on embodiment. It is a figure explaining the process which produces | generates the blend area | region which concerns on embodiment. It is a figure explaining the process which produces | generates the blend area | region which concerns on embodiment. It is a figure explaining the process which produces | generates the blend area | region which concerns on embodiment. It is a figure explaining the process which produces | generates the blend area | region which concerns on embodiment. It is a figure explaining the example of a display of an object and a blend area | region.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<System configuration>
First, the system configuration of the display system 1 according to the present embodiment will be described with reference to FIG. 1A. FIG. 1A is a diagram illustrating an example of a system configuration of a display system 1 according to the embodiment.

  As illustrated in FIG. 1A, the display system 1 according to the embodiment is mounted on a moving body such as a vehicle, a ship, an aircraft, a personal mobility, and an industrial robot. The display system 1 includes a display device 10 and a camera 20. In the following, an example in which the display system 1 is mounted on a vehicle will be described, but the display system 1 can also be applied to a moving body other than the vehicle. The vehicle includes, for example, an automobile, a motorbike, a light vehicle, and a railway vehicle.

  The display device 10 is, for example, a display device such as a HUD or a head mounted display. Below, the case where the display apparatus 10 is HUD which displays a virtual image is demonstrated as an example. The display device 10 is installed in a dashboard of the vehicle 301, for example. The projection light L, which is image light emitted from the display device 10, is reflected by the windshield 302 as a light transmission / reflection member and travels toward the passenger 300 who is a viewer. Here, the transmission / reflection member is, for example, a member that transmits part of light and reflects part of light. As a result, the image is projected onto the windshield 302, and the passenger 300 can visually recognize objects (contents) such as navigation graphics, characters, icons, etc. in an environment outside the vehicle 301. A combiner as a transmission / reflection member may be installed on the inner wall surface or the like of the windshield 302 so that the driver can visually recognize the virtual image by the projection light L reflected by the combiner. FIG. 1B is a diagram illustrating an example of a range in which an image is projected by the display device 10 according to the embodiment. As illustrated in FIG. 1B, the display device 10 projects an image on a projection range 303 on the windshield 302, for example.

  The camera 20 is a camera that captures an image of the front of the vehicle 301 or the like. The camera 20 may be provided on the top of the windshield 302, for example. The camera 20 may be a camera for measuring the distance between the vehicle ahead and the vehicle 301 for automatic driving.

<Hardware configuration>
Next, the hardware configuration of the display device 10 according to the present embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the display device 10 according to the embodiment.

  The display device 10 includes an FPGA 251, a CPU (Central Processing Unit) 252, a ROM 253, a RAM 254, an interface (hereinafter referred to as I / F) 255, a bus line 256, an LD driver 257, a MEMS controller 258, and an auxiliary storage device 259. Yes. The FPGA 251 controls the operation of the laser light sources 201R, 201G, and 201B of the light source unit 220 by the LD driver 257, and controls the operation of the MEMS 208a of the optical scanning device by the MEMS controller 258. The CPU 252 controls each function of the display device 10. The ROM 253 stores various programs such as a program (image processing program) executed by the CPU 252 to control each function of the display device 10.

  The RAM 254 reads and stores the program from the ROM 253 or the auxiliary storage device 259 when there is an instruction to start the program. The CPU 252 realizes functions related to the display device 10 according to a program stored in the RAM 254.

  The I / F 255 is an interface for communicating with an external controller or the like. For example, the I / F 255 is connected to a vehicle navigation device, various sensor devices, or the like via a CAN (Controller Area Network) of the vehicle 301. The I / F 255 is connected to a camera 20 that captures an image through the windshield 302.

  The display device 10 can read and write the recording medium 255a via the I / F 255. An image processing program for realizing processing in the display device 10 may be provided by the recording medium 255a. In this case, the image processing program is installed in the auxiliary storage device 259 from the recording medium 255a via the I / F 255. However, it is not always necessary to install the image processing program from the recording medium 255a, and the image processing program may be downloaded from another computer via a network. The auxiliary storage device 259 stores the installed image processing program and also stores necessary files and data.

  An example of the recording medium 255a is a portable recording medium such as a flexible disk, a CD-ROM, a DVD disk, an SD memory card, or a USB (Universal Serial Bus) memory. An example of the auxiliary storage device 259 is an HDD (Hard Disk Drive) or a flash memory. Both the recording medium 255a and the auxiliary storage device 259 correspond to computer-readable recording media. In FIG. 2, a portion including the CPU 252, ROM 253, RAM 254, I / F 255, bus line 256, and auxiliary storage device 259 can also be referred to as an image processing device or an information processing device (computer).

<Functional configuration>
Next, the functional configuration of the display device 10 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of functional blocks of the display device 10 according to the embodiment.

  The display device 10 includes an acquisition unit 11, a control unit 12, a generation unit 13, and a display unit 14. Each of these units is realized by processing that one or more programs installed in the display device 10 cause the CPU 252 of the display device 10 to execute.

  The acquisition unit 11 acquires an image such as the front of the vehicle 301 captured by the camera 20.

  The control unit 12 guides a route from the current position of the vehicle 301 to a preset destination. For example, the control unit 12 determines an object indicating the traveling direction of the route, such as a right turn, a left turn, and a straight turn, and displays the object in a manner such that the object is placed on the road. To display. In addition, the control unit 12 displays a number indicating the vehicle speed (speed) of the vehicle 301 on a background in front of the vehicle 301. Here, the display device 10 may acquire information on the vehicle speed of the vehicle 301 from an ECU (Electronic Control Unit) of the vehicle 301, for example.

  When an object to be displayed is determined by the control unit 12, the generation unit 13 generates (determines) an image to be displayed in an area around the object according to the image acquired by the acquisition unit 11.

  The display unit 14 displays the object determined by the control unit 12 and the image of the area around the object generated by the generation unit 13.

<Processing>
Next, processing of the display device 10 according to the embodiment will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of processing of the display device 10 according to the embodiment. Note that the process of FIG. 4 may be executed for each frame included in a moving image shot by the camera 20, for example.

  In step S <b> 1, the acquisition unit 11 acquires an image (frame) such as the front of the vehicle 301 captured by the camera 20. Subsequently, the control unit 12 determines an object to be displayed. Here, for example, the control unit 12 determines an object for guiding a route from the current position of the vehicle 301 to a preset destination (step S2).

  Subsequently, based on the acquired image and the object, the generation unit 13 generates an image of a blend area that is an area for displaying an image corresponding to the image of the camera 20 around the object (step S3). ). Subsequently, the display unit 14 displays the object and the blend area (step S4), and ends the process.

<< Display object generation process >>
Next, with reference to FIGS. 5 to 6E, an example of a process for generating an image of the blend area in step S3 of FIG. FIG. 5 is a flowchart illustrating an example of processing for generating an image of a blend area according to the embodiment. 6A to 6E are diagrams for describing processing for generating a blend area according to the embodiment.

  In step S <b> 101, the generation unit 13 calculates a blend region that is an area around the object to be displayed and is an enlarged shape of the object. Here, in the example of FIG. 6A, the generation unit 13 calculates a blend region 602 having a shape obtained by enlarging the region 601 of the object around the region 601 of the object.

  Subsequently, the generation unit 13 smoothly deforms the corners of the calculated blend area (step S102). Here, in the example of FIG. 6A, the generation unit 13 calculates the blend region 603 in which the corners that are the portions of the vertices are rounded by replacing the contours of the vertex portions of the blend region 602 with arcs. Yes. Note that the generation unit 13 may smooth the corner by another known method.

  Subsequently, the generation unit 13 changes the length of the blend area in the front direction of the vehicle 301 (step S103). In the example of FIG. 6B, the generation unit 13 calculates the blend region 604 in which the length of the blend region 603 in the vertical direction (vertical direction, forward direction of the vehicle 301) is changed. Here, the generation unit 13 may increase the length as the vehicle speed of the vehicle 301 increases. This is because it is considered that the higher the vehicle speed of the vehicle 301, the more noticeable the object is with respect to the passenger.

  In addition, the generation unit 13 recognizes the image of the camera 20, and when the road on which the vehicle 301 is traveling is a gravel road or the like in which the road pattern is relatively conspicuous, the longer the vehicle speed of the vehicle 301, the longer The length may be shortened. Due to the delay for generating an image around the object based on the image of the camera 20, the higher the vehicle speed of the vehicle 301, the faster the moving speed of the actual road pattern with respect to the occupant and the object displayed around the object. The difference from the movement speed of the road pattern in the image becomes large. For this reason, it is considered that as the vehicle speed of the vehicle 301 increases, the image around the object becomes more conspicuous to the passenger.

  Further, the generation unit 13 may increase the length as the distance from the vehicle 301 to a position such as an actual road on which the object is superimposed is displayed. Thus, as the distance increases, the ratio value (L1 / L2) between the length (L1) of the area around the predetermined object and the length (L2) of the object in the forward direction of the vehicle 301 is increased. ) Is increased. This is because, when the surface of the object and the blend area is placed on a background object by the process of step S105 described later, the area of the surface of the blend area that can be seen by the passenger becomes smaller as the distance is longer. is there.

  Subsequently, the generation unit 13 sets so as to blur pixels around the blend area (step S104). Here, for example, the generation unit 13 may set the pixel density smaller as the distance from the object increases in the blend region. In addition, for example, the generation unit 13 may set the blend region to have higher transparency by lowering the luminance of the pixel displayed in the HUD as the distance from the object increases in the blend region. Good. In the example of FIG. 6C, the generation unit 13 further decreases the pixel density and the pixel luminance as the distance from the nearest coordinates included in the object region 601 increases in the blend region 604 of FIG. 6B. .

  Subsequently, the generation unit 13 looks like the surface of the object and the blend area is placed on the background object at the position where the object is displayed as seen from the passenger of the vehicle 301. The object and the blend area are deformed (step S105).

  In FIG. 6D, when viewed from the side of the vehicle 301, the eye 610 of the passenger 300, the windshield 302, a region 612 in which a virtual image is displayed by the display device 10 (projection range 303 in FIG. 1B), and a region The position of an area 612A and the like where 612 is superimposed on an object in the background is illustrated. FIG. 6D shows a region 611 in which the region 601 of the object in the region 612 is deformed, and a blend region 613 in which the blend region 604 in the region 612 is deformed. FIG. 6D shows a region 611A and a region 613A, which are regions where the object region 611 and the blend region 613 overlap each other on an object such as a background road. In the example of FIG. 6D, the distance from the eye 610 of the passenger 300 to the area 611 and the blend area 613 of the object is about 4 m. The distance from the eye 610 of the passenger 300 to the area 611A and the area 613A on the road is about 20 m.

  In the process of step S105, the generation unit 13 transforms the blend region 604 including the region 601 in FIG. 6C into a shape laid on a background road or the like as illustrated in FIG. 6D. Here, when the region 613A on the road in FIG. 6D is viewed from directly above in the vertical direction, the generation unit 13 assumes that the region 601 and the blend region 604 are displayed in the shape of FIG. The shape of the area 613A when the area 613A is viewed from the position of the eye 610 of the passenger 300 is calculated.

  Then, the generation unit 13 transforms the region 601 and the blend region 604 in FIG. 6C into a region 611 and a blend region 613 in FIG. 6E, respectively. The generation unit 13 deforms the region 601 and the blend region 604 in FIG. 6C into a shape in which the height in the vertical direction is reduced and the width in the horizontal direction is further reduced as the vehicle 301 travels upward. By doing so, the area 611 and the blend area 613 in FIG. 6E are respectively calculated.

  For example, the generation unit 13 sets a position such as the height of the vehicle 301 of the passenger 300 from the road of the vehicle 301 to a preset position, and sets the road in front of the vehicle 301 to be flat. The shape of the region 613A on the road or the like when viewed from the position of the eye 610 may be calculated geometrically. Note that the generation unit 13 may recognize the position of the eye 610 of the passenger 300 based on the image of the in-vehicle camera. The generation unit 13 may recognize the height of the road ahead of the vehicle 301 based on an image from a stereo camera or the like.

  Subsequently, the generation unit 13 combines the image generated based on the image of the camera 20 with the deformed blend region (step S106), and ends the process. Here, as illustrated in FIG. 6D, the generation unit 13 may extract an image in the area 613A on the road or the like from the image of the camera 20 and synthesize the image in the blend area 613 in FIG. 6E. It is assumed that the position where the camera 20 is attached is set in the display device 10 in advance. When the position of the camera 20 and the eye 610 of the occupant 300 are separated from each other by a predetermined threshold value or more, the generation unit 13 boards the image in the area 613A on the road or the like photographed by the camera 20. The image may be synthesized in the blend region 613 in FIG. 6E after being geometrically deformed so as to be an image of the region 613A viewed from the position of the eye 610 of the person 300. Further, the generation unit 13 may combine an image of pixels having an average value such as luminance of an image in the area 613A on the road or the like extracted from the image of the camera 20 into the blend area 613 in FIG. 6E. Further, the generation unit 13 may combine an image of a region ahead by a predetermined distance of a region 613A on the road or the like as viewed from the vehicle 301 into the blend region 613 in FIG. 6E. In this case, the generation unit 13 may set the predetermined distance according to a delay time from when an image is captured by the camera 20 until the predetermined distance is displayed in the blend area, and the vehicle speed of the vehicle 301.

  FIG. 7 is a diagram illustrating a display example of objects and blend areas. In the example of FIG. 7, in the region 612 where the virtual image is displayed by the display device 10, the object region 611 and the blend region 613 of FIG. Is displayed in front of. Thereby, it becomes difficult for a passenger to recognize the boundary between an object such as a route guidance graphic and the background of the object, and the sense of incongruity can be reduced.

<Summary of Embodiment>
An object displayed by HUD or the like has a light source color because it is displayed by light from the light source, and a background of the object is an object color by reflection of sunlight or the like. Further, the distance from the passenger's eyes to the virtual or real image of the object is different from the distance from the passenger's eyes to the real object in the background of the object. For this reason, even if the display position of the object is geometrically matched with the position of the actual object in the background, the object may not be easily perceived as existing in the actual environment of the background.

  According to the above-described embodiment, the display device that displays a predetermined object on the environment outside the moving body displays an image generated according to the image outside the moving body around the predetermined object. . Thereby, the uncomfortable feeling of the object displayed in addition to the actual environment can be reduced.

<Others>
Each functional unit of the display device 10 may be realized by cloud computing including one or more computers. Moreover, you may comprise at least 1 function part among each function parts of the display apparatus 10 as an apparatus different from the apparatus which has another function part. In this case, for example, the control unit 12 may be included in a vehicle-mounted or portable navigation device or a server device on the cloud. That is, the display device 10 includes a form constituted by a plurality of devices. The generation unit 13 is an example of a “determination unit”.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

DESCRIPTION OF SYMBOLS 1 Display system 10 Display apparatus 11 Acquisition part 12 Control part 13 Generation part 14 Display part 20 Camera 301 Vehicle 302 Windshield

JP 2017-142491 A

Claims (9)

A display device for displaying a predetermined object at a position overlapping with an environment outside a moving body,
An acquisition unit for acquiring an image outside the moving body;
A determination unit that determines an image of a region around the predetermined object according to an image acquired by the acquisition unit;
And a display unit configured to display the predetermined object and an image of an area around the predetermined object determined by the determination unit. The determining unit sets the shape of the image of the area around the predetermined object to a shape in which the predetermined object is enlarged and the corners are smoothed.
The display device according to claim 1. The determining unit determines an image of an area around the predetermined object so that at least one of pixel luminance and pixel density differs according to a distance from the predetermined object.
The display device according to claim 1. The determining unit sets an image of a region around the predetermined object to a shape in which a surface of the region is placed on a background object of the predetermined object.
The display apparatus as described in any one of Claims 1 thru | or 3. The determining unit changes the length of the area around the predetermined object in the forward direction of the moving body according to the speed of the moving body.
The display apparatus as described in any one of Claims 1 thru | or 4. The determining unit is configured to determine a length of the predetermined object and a length of a region around the predetermined object in a forward direction of the moving body according to a distance from the moving body to a background object of the predetermined object. Change the ratio,
The display device according to any one of claims 1 to 5. A moving object,
A camera that captures an image of the outside of the moving body, a transmission / reflection member, and a display device that displays a predetermined object at a position overlapping the environment outside the moving body;
The display device
An acquisition unit for acquiring an image of the camera;
A determination unit that determines an image of a region around the predetermined object according to an image acquired by the acquisition unit;
And a display unit that displays the predetermined object and an image of an area around the predetermined object determined by the determination unit, using the transmission / reflection member. A display device that displays a predetermined object at a position overlapping with the environment outside the moving body,
Processing to acquire an image outside the moving body;
A process of determining an image of an area around the predetermined object according to the image acquired by the process of acquiring;
A process of displaying the predetermined object and an image of an area around the predetermined object determined by the determining process;
Display method to execute. On the computer,
Processing to acquire an image outside the moving body;
A process of determining an image of a region around a predetermined object according to the image acquired by the acquisition process;
A program for executing a process of displaying the predetermined object and an image of an area around the predetermined object determined by the determining process at a position overlapping with an environment outside the moving body.

Images