JP7145509B2 - VIDEO DISPLAY SYSTEM, VIDEO DISPLAY METHOD, PROGRAM AND MOBILE BODY - Google Patents

Description

TECHNICAL FIELD The present disclosure relates to a video display system, a video display method, a program, and a moving body provided with the video display system.

As an example of a video display system, for example, there is a head-up display device (for example, Patent Literature 1). The head-up display device disclosed in Patent Literature 1 irradiates the windshield of a vehicle with display light output from a display device, thereby allowing the user to superimpose a virtual image of an image displayed on the display device on the actual scene in front of the vehicle. can be visually recognized.

The head-up display device disclosed in Patent Literature 1 includes a vehicle vibration input section and a display control section. Vehicle vibration information is input to the vehicle vibration input unit. The display control unit corrects the display position of the virtual image by correcting the display image based on the vibration information. More specifically, after drawing the display image on the first layer, the display control unit moves the display position of the display image drawn on the first layer in a direction that cancels out the vibration of the vehicle. Note that the first layer is a surface provided virtually corresponding to the display surface of the display. As a result, even if the vehicle vibrates, the display position of the virtual image is corrected so as to maintain a predetermined positional relationship with respect to the object in the real scene.

JP 2017-13590 A

However, in the head-up display device disclosed in Patent Document 1, the correction of the display position of the display image based on the vibration information is executed in the drawing processing stage of drawing the display image, so the display delay of the display unit (for example, several frame delay) is likely to occur.

The present disclosure has been made in view of the circumstances described above, and an object thereof is to provide a video display system, a video display method, a program, and a moving object that can reduce display delay of a display image.

A video display system or the like according to an aspect of the present disclosure is a video display system that allows a viewer to visually recognize a virtual image of a display image displayed by a display device by superimposing it on a real scene in front of a moving object, wherein An image consisting of a first area that is an area that does not move over time and a second area that is an area that moves over time is drawn, and a signal indicating the first area and a signal indicating the second area are output in this order. a drawing unit, and during a period in which the first region is being acquired from the drawing unit, vibration information indicating the vibration amount of the moving body is acquired, and for the second region being acquired from the drawing unit, the Based on the vibration information, the vibration correction processing is executed, and the second region subjected to the vibration correction processing and the first region obtained from the drawing unit are combined in this order to form the display image. and a display device output unit for outputting to a display device.

In addition, some specific aspects of these may be realized using a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. It may be implemented using any combination of integrated circuits, computer programs and storage media.

Advantageous Effects of Invention According to the present disclosure, it is possible to realize a video display system or the like capable of reducing display delay of a display image.

FIG. 1 is a conceptual diagram of a moving object including a video display system according to an embodiment. FIG. 2 is a conceptual diagram showing the configuration of the HUD device included in the moving object shown in FIG. FIG. 3 is a conceptual diagram showing a user's field of view when using the HUD device shown in FIG. 4 is a block diagram showing an example of a detailed configuration of the display device shown in FIG. 2. FIG. FIG. 5 is a block diagram showing an example of the detailed configuration of the video display system according to the embodiment. 6A is a diagram showing a video format of a display image output by the display device output unit according to the embodiment; FIG. FIG. 6B is a diagram showing a video format of a drawn image output by the drawing unit according to the embodiment. 7 is a block diagram illustrating an example of a detailed configuration of a drawing unit according to the embodiment; FIG. 8 is a block diagram illustrating an example of a detailed configuration of a display device output unit according to the embodiment; FIG. FIG. 9 is a diagram for explaining a detailed example of vibration correction processing according to the embodiment. FIG. 10 is a diagram for explaining a detailed example of composition processing according to the embodiment. FIG. 11 is a flow chart showing operation processing executed by the video display system according to the embodiment. FIG. 12 is a diagram showing an example of video formats and timings output by the drawing unit and the display device output unit according to the present embodiment. FIG. 13 is a diagram showing a comparative example. FIG. 14 is an explanatory diagram of an effect according to the embodiment.

A video display system according to one aspect of the present disclosure is a video display system that allows a viewer to visually recognize a virtual image of a display image displayed by a display device by superimposing it on a real scene in front of a moving body. An image composed of a first area that does not move and a second area that moves with time is drawn, and a signal indicating the first area and a signal indicating the second area are output in this order. and acquiring vibration information indicating a vibration amount of the moving body during the period in which the first region is acquired from the drawing unit, and applying the vibration to the second region being acquired from the drawing unit. Based on the information, vibration correction processing for correcting the display position of the virtual image of the second region is executed, and the second region subjected to the vibration correction processing and the first region acquired from the drawing unit are combined into this a display device output unit that combines the images in order and outputs them to the display device as the display image.

With this configuration, correction of the display position of the virtual image according to the vibration information indicating the amount of vibration of the moving object is performed by the display device output unit located after the drawing unit and closer to the display device. display delay can be reduced. Furthermore, since the drawing unit outputs the signal indicating the first area and the signal indicating the second area in this order, the display device output unit can acquire the vibration information while acquiring the signal indicating the first area. . As a result, the display device output unit outputs the signal indicating the second area and the signal indicating the first signal in this order as the correction amount used for executing the correction processing for correcting the display position of the virtual image in the second area. It can be determined more quickly from the vibration information as compared with the case of outputting it to the drawing unit. Therefore, the delay from obtaining the vibration information to executing the correction process for correcting the display position of the virtual image in the second area can be further reduced, so that the display delay of the image corresponding to the virtual image can be further reduced. .

Here, for example, the display device output unit outputs an image obtained by synthesizing the second region on which the vibration correction processing has been performed and the first region acquired from the drawing unit in this order, and the vibration correction processing is A different distortion correction process may be further performed and output as the display image to the display device.

According to this configuration, the display device output section can perform the vibration correction process and the distortion correction process, and the processing efficiency can be improved as compared with the case where separate processing sections are used.

Further, for example, the display device output unit may combine the second region on which the vibration correction processing has been performed and the first region acquired from the drawing unit in this order, and may generate an image in which the vibration correction processing is different from the vibration correction processing. A distortion correction process may be further performed and output to the display device as the display image.

Further, a mobile object according to an aspect of the present disclosure includes the image display system according to any one of the above aspects.

According to this configuration, it is possible to implement the image display system having the above effects on a moving body.

Further, a video display method according to an aspect of the present disclosure provides video display using a video display system that allows a viewer to visually recognize a virtual image of a display image displayed by a display device by superimposing it on a real scene in front of a moving object. A method of rendering an image consisting of a first region, which is a region that does not move over time, and a second region, that is a region that moves over time, and showing a signal indicative of the first region and the second region. a drawing step of outputting signals in this order; obtaining vibration information indicating a vibration amount of the moving object during a period in which the signal indicating the first region is obtained; and obtaining the vibration information in a period following the period. Vibration correction processing for correcting the display position of the virtual image of the second region is performed on the second region based on the vibration information, and the second region and the period in which the vibration correction processing is performed are acquired. and an output step of synthesizing the obtained first regions in this order and outputting them as the display image to the display device.

Further, a program according to one aspect of the present disclosure is a program for causing a computer to execute the video display method described in the aspect above.

It should be noted that some specific aspects of these may be implemented using a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. It may be implemented using any combination of integrated circuits, computer programs or storage media.

Hereinafter, a video display device and the like according to one aspect of the present disclosure will be specifically described with reference to the drawings. It should be noted that each of the embodiments described below is a specific example of the present disclosure. Numerical values, shapes, materials, components, arrangement positions of components, and the like shown in the following embodiments are examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims representing the highest concept will be described as arbitrary constituent elements. Moreover, each content can also be combined in all the embodiments.

(Embodiment)
(1) Overview FIG. 1 is a conceptual diagram of a moving object 100 including a video display system 10 according to an embodiment. FIG. 2 is a conceptual diagram showing the configuration of the HUD device 10A included in the moving object 100 shown in FIG. FIG. 3 is a conceptual diagram showing the field of view of the user 200 when using the HUD device 10A shown in FIG.

Although the moving body 100 is described below as being an automobile, it may be something other than an automobile. That is, the mobile object 100 is not limited to an automobile, and may be, for example, a two-wheeled vehicle, a train, an aircraft, a construction machine, a ship, or the like.

A HUD (Head-Up Display) device 10A superimposes a virtual image of a display image displayed by a display device on a real scene in front of the moving object 100 so that the viewer can see it. In this embodiment, the HUD device 10A is composed of a main body 1 and an image display system 10, as shown in FIG.

The image display system 10 is included in, for example, a moving body 100 and constitutes a HUD device 10A provided in the moving body 100, as shown in FIGS. Note that the image display system 10 may include part or all of the main unit 1 . When the image display system 10 includes the entire body section 1, the image display system 10 can also be called a HUD device 10A.

The main body 1 is arranged in the vehicle interior of the moving body 100 so as to project a display image onto the windshield 101 of the moving body 100 from below. In the example shown in FIG. 1 , the main body 1 is arranged inside the dashboard 102 below the windshield 101 . When the display image is projected onto the windshield 101 from the main unit 1, the user 200 such as the driver sets the display image projected onto the windshield 101 to the front of the moving body 100 (outside the moving body). It is viewed as a virtual image 300 displayed in the target space 400 .

The term "virtual image" as used herein means an image formed by the divergent rays of light that appears to be an actual object when the light of the display image emitted from the main unit 1 diverges from a reflecting object such as the windshield 101. do. Also, the windshield 101 is a projection target onto which a display image is projected.

Therefore, the user 200 driving the moving object 100 can see the virtual image 300 projected by the main unit 1 superimposed on the real scene spreading in front of the moving object 100, as shown in FIG. Therefore, according to the HUD device 10A, it is possible to display various types of driving support information as the virtual image 300 superimposed on the real scene so that the user 200 can visually recognize it. Driving support information includes, for example, vehicle speed information, navigation information, pedestrian information, forward vehicle information, lane deviation information, and vehicle condition information. As a result, the user 200 can visually acquire the driving support information only by slightly moving the line of sight from the state in which the line of sight is directed forward of the windshield 101 .

The virtual image 300 displayed in the target space 400 is, for example, information indicating the traveling direction of the mobile object 100 as navigation information, and is an arrow indicating a right turn or left turn on the road surface 600 . This kind of virtual image 300 is an image displayed using an augmented reality technology, and is located at a specific position in the real scene (road surface 600, buildings, surrounding vehicles, pedestrians, etc.) seen by the user 200. It is superimposed and displayed.

A virtual image 300 displayed in the target space 400 is formed on a virtual plane 501 that intersects the optical axis 500 of the main body 1 of the HUD device 10A, as shown in FIG. The optical axis 500 is along the road surface 600 in front of the moving body 100 in the target space 400 in front of the moving body 100 . A virtual plane 501 on which the virtual image 300 is formed is substantially perpendicular to the road surface 600 . For example, if the road surface 600 is horizontal, the virtual image 300 is displayed along the vertical plane.

In the present embodiment, the main body 1 includes a display device 2 including a display 20 and a display control section 21, and a projection section 3, as shown in FIG.

The display control section 21 controls the projection section 3 and the display device 20 . The display device 20 has a display surface 20a and displays a display image 700 on the display surface 20a. The display device 20 emits display light indicating the display image 700 toward the first mirror 31 of the projection unit 3 .

The projection unit 3 performs a projection process of projecting a virtual image 300 corresponding to a display image 700 onto a target space 400 using the display light of the display device 20, as shown in FIG. 1, for example.

When the main body 1 is mounted on the moving body 100, the attitude of the main body 1 changes along with the attitude of the moving body 100 due to vibrations caused by, for example, the condition of the road surface 600 and the acceleration/deceleration of the moving body 100. Change. For example, if the moving body 100 is tilted forward, the main body 1 is also tilted forward, and if the body 100 is tilted backward, the main body 1 is also tilted backward. Therefore, when the posture of the main body 1 changes, the relative arrangement relationship between the virtual image 300 and the real scene changes. In other words, when the posture of the main body 1 changes, the virtual image 300 is displayed at a position shifted from the specific position where it should be superimposed in the actual scene seen by the user 200 . Note that the vibration that changes the posture of the moving body 100 includes not only vibration caused by acceleration and deceleration of the moving body 100, but also vibration that causes vertical movement of the moving body 100 when the moving body 100 runs on an uneven road. mentioned.

Therefore, in the video display system 10 , the display position of the virtual image 300 is corrected according to the posture change of the moving object 100 . As a result, the video display system 10 can superimpose and display the virtual image 300 at a specific position where the virtual image 300 should be superimposed in the actual scene viewed from the user 200 even if the posture of the moving object 100 changes.

Of the vibrations that change the posture of the moving body 100, the vibration that causes vertical movement of the moving body 100 when the moving body 100 runs on an uneven road is less than the vibration caused by the acceleration/deceleration of the moving body 100. is a shorter time (period) oscillation. Therefore, the correction processing of the display position of the virtual image 300 is required to be performed in real time.

In the image display system 10, at a processing stage closer to the output processing stage to the display device 2, vibration information indicating the vibration amount of the moving body 100 is acquired and correction processing is executed. That is, in the image display system 10, the time from acquisition of vibration information to execution of correction processing for correcting the display position of the virtual image is shortened. As a result, the delay in the processing for correcting the display position of the virtual image 300 can be shortened, and even when the processing for correcting the display position of the virtual image 300 requires real-time performance, it is possible to respond.

(2) Configuration [Configuration of main unit 1]
The main body 1 is fixed inside a dashboard 102 of a mobile body 100, as shown in FIG. 1, for example. The main body 1 is composed of, for example, a housing, but it may be a frame, a plate, or the like instead of the housing. As described above, the main unit 1 includes the display device 2 including the display device 20 and the display control unit 21 and the projection unit 3 as shown in FIG.

The projection unit 3 executes projection processing for projecting the display light of the display device 20 onto the target space 400 . In the present embodiment, as shown in FIG. 1, the projection unit 3 projects display light representing a display image 700 onto the windshield 101, thereby projecting a virtual image 300 corresponding to the display image 700 onto the target space 400. .

In this embodiment, the projection unit 3 has a first mirror 31 and a second mirror 32, as shown in FIG. The first mirror 31 is, for example, a convex mirror, and the second mirror 32 is, for example, a concave mirror. The first mirror 31 and the second mirror 32 are arranged in the order of the first mirror 31 and the second mirror 32 on the optical path of the display light of the display 20 . More specifically, the first mirror 31 is arranged in front of the display surface 20a of the display 20 so that the display light of the display 20 is incident thereon. The first mirror 31 reflects the display light of the display 20 toward the second mirror 32 . The second mirror 32 is arranged at a position such as a position in front and below the first mirror 31 where the display light of the display 20 reflected by the first mirror 31 is incident. The second mirror 32 reflects the display light of the display 20 reflected by the first mirror 31 upward, that is, toward the windshield 101 .

With this configuration, the projection unit 3 enlarges or reduces the display image 700 displayed on the display surface 20a of the display device 20 to an appropriate size and projects it onto the windshield 101 as a projection image. As a result, the virtual image 300 is displayed in the target space 400 . That is, the virtual image 300 of the display image 700 projected from the HUD device 10A is superimposed on the real scene spreading in front of the mobile body 100 within the visual field of the user 200 driving the mobile body 100 .

FIG. 4 is a block diagram showing an example of the detailed configuration of the display device 2 shown in FIG. 2. As shown in FIG.

As shown in FIGS. 2 and 4, the display device 2 includes a display device 20 and a display control unit 21, executes display processing for displaying a display image 700 on the display surface 20a, and projects the displayed display image 700 to a projection unit. Aim at 3. In the example shown in FIG. 3, the display image 700 includes, for example, an area representing an arrow displayed on the road surface on which the moving object 100 travels, and an area representing numbers displaying the speed of the moving object 100. It is an image for guiding a driving route to 100 destinations. The virtual image 300 includes a virtual image 300a representing, for example, an arrow and a virtual image 300b representing, for example, velocity (see FIG. 3).

As shown in FIG. 4, the display device 20 includes a liquid crystal panel 201 and a light source device 202, and emits display light indicating a display image 700 displayed on the display surface 20a toward the first mirror 31 of the projection unit 3. .

The liquid crystal panel 201 displays a display image 700 formed by the video display system 10 . The liquid crystal panel 201 is, for example, an LCD (Liquid Crystal Display). The liquid crystal panel 201 is arranged in front of the light source device 202 . The front surface of the liquid crystal panel 201 (that is, the surface opposite to the light source device 202) constitutes the display surface 20a. A display image 700 is displayed on the display surface 20a.

The light source device 202 irradiates substantially the entire rear surface of the liquid crystal panel 201 with light using solid-state light-emitting elements such as light-emitting diodes or laser diodes. The light source device 202 illuminates a display image 700 displayed on the liquid crystal panel 201 in front of the liquid crystal panel 201 . Thus, the light source device 202 is used as a backlight for the liquid crystal panel 201 .

The display control unit 21 drives the liquid crystal panel 201 based on the display image 700 output from the video display system 10 to display the display image 700 on the display surface 20a. The display control unit 21 also turns on the light source device 202 to illuminate the display image 700 displayed on the liquid crystal panel 201 in front of the liquid crystal panel 201 . The light illuminated in front of the liquid crystal panel 201 at this time is the light reflecting the display image 700 displayed on the liquid crystal panel 201 . Therefore, the image displayed on the liquid crystal panel 201 is projected in front of the liquid crystal panel 201 by the display light from the light source device 202 .

The display control unit 21 is composed of, for example, a microcomputer mainly composed of a CPU (Central Processing Unit) and a memory. In other words, the display control unit 21 is realized by a computer having a CPU and memory, and the computer functions as the display control unit 21 by the CPU executing a program stored in the memory. The program is pre-recorded in the memory of the display control unit 21, but may be provided through an electric communication line such as the Internet or recorded in a recording medium such as a memory card.

[Configuration of video display system 10]
FIG. 5 is a block diagram showing an example of the detailed configuration of the image display system 10 according to this embodiment.

The video display system 10 acquires driving assistance information. In the following description, the video display system 10 acquires navigation information from a navigation device mounted on the mobile object 100. FIG.

Here, the navigation information is, for example, information such as the travel route to the destination to which the mobile object 100 is headed and the distance to the nearest intersection. Then, when the image display system 10 determines that the moving object 100 has approached the nearest intersection on the travel route within a certain distance based on the acquired navigation information, the image display system 10 displays an arrow for guiding the traveling direction at the intersection. A display image 700 containing an image is formed. At that time, the arrow image of the display image 700 is formed so that the virtual image 300a thereof is displayed at a predetermined position 401 within the target space 400 (for example, a position overlapping along the intersection). The video display system 10 outputs the formed display image 700 to the display device 2 .

In this embodiment, the video display system 10 includes a drawing section 11 and a display device output section 12, as shown in FIG.

<Drawing unit 11>
FIG. 6A is a diagram showing the video format of display image 700 output by display device output unit 12 according to the present embodiment. FIG. 6B is a diagram showing the video format of the drawn image 701 output by the drawing unit 11 according to this embodiment.

The drawing unit 11 draws an image composed of a first region (ST region) that is a region that does not move over time and a second region (AR region) that is a region that moves over time. ) and a signal indicating the second area (AR area) are output in this order.

Hereinafter, the first area will be referred to as an ST area (Graphical Area of Static Information), and the second area will be referred to as an AR area (Graphical Area of Augmented Reality). The ST area is an area for showing various types of information such as the meter, speed, fuel system, etc. of the moving body 100, that is, basic information about the moving body 100, and is an area that does not move over time. In other words, the ST area is an area whose virtual image does not have a predetermined positional relationship with respect to a specific object in the real scene. Therefore, as shown in FIG. 6A, the ST area is displayed at a specific lower position in the display image 700, but is an area that does not cause discomfort to the user 200 even if the position shifts due to vibration.

The AR area includes, for example, an image to warn of a collision with an object in front of the mobile object 100, an image such as an arrow displayed on the road surface to guide the travel route to the destination, and an image of the travel lane. A region that shows real-time content, such as images that are displayed to suppress deviations. In other words, the AR area is an area where real-time content is superimposed and displayed, and is an area that moves over time. In other words, the AR area is an area whose virtual image has a predetermined positional relationship with respect to a specific object in the real scene. Therefore, as shown in FIG. 6A, the AR area is displayed at a specific position above the display image 700 and superimposed on the specific object, and if the position shifts due to vibration, the user 200 feels uncomfortable. This is the area where the

In the present embodiment, as shown in FIG. 6A, the display image 700 is output by the display device output unit 12 in a video format in which a signal indicating the AR area and a signal indicating the ST area are configured in this order. On the other hand, the drawn image 701 is output by the drawing unit 11 in a video format in which a signal indicating the ST area and a signal indicating the AR area are configured in this order. That is, the drawing unit 11 outputs the drawn image 701 to the display device output unit 12 in a video format opposite to the video format output by the display device output unit 12 to the display device 2 .

FIG. 7 is a block diagram showing an example of the detailed configuration of the drawing unit 11 according to this embodiment.

In addition, in the present embodiment, the rendering unit 11 includes a rendering unit body 111 and a rendering buffer 112, as shown in FIG.

The drawing unit main body 111 first positions the ST area indicating the basic information of the moving body 100 below in the three-dimensional virtual space corresponding to the target space 400, and then lowers the AR area indicating the real-time content acquired by the video display system 10. is positioned above the three-dimensional image. Here, the real-time content is, for example, an arrow for guiding the direction of travel at an intersection based on the navigation information obtained by the video display system 10 . The basic information of the moving body 100 is, for example, the speed of the moving body 100 . Next, the drawing unit main body 111 converts the drawn three-dimensional image into a two-dimensional image by projecting it onto a predetermined projection plane in the three-dimensional virtual space, and uses this two-dimensional image as a drawn image. In this way, the rendering unit main body 111 renders a rendered image with the ST area positioned below and the AR area positioned above.

Then, the drawing unit main body 111 temporarily stores in the drawing buffer 112 not the drawn image but the drawn image 701 formed by positioning the ST area in the drawn image above and the AR area below. .

The rendering buffer 112 temporarily stores a rendered image 701 formed by the rendering unit body 111 . The drawing buffer 112 then outputs the drawing image 701 to the display device output unit 12 according to the video format shown in FIG. 6B. More specifically, the drawing buffer 112 first outputs a signal indicating the ST region to the display device output unit 12, and then outputs a signal indicating the AR region to the display device according to the video format shown in FIG. 6B. Output to the output unit 12 .

<Display device output unit 12>
The display device output unit 12 executes various processes on the drawn image 701 acquired from the drawing unit 11 and outputs the display image 700 obtained by this processing to the display device 2 .

More specifically, the display device output unit 12 acquires vibration information indicating the amount of vibration of the moving body 100 while acquiring the first area from the drawing unit 11 . Further, the display device output unit 12 performs vibration correction processing for correcting the display position of the virtual image of the second region based on the vibration information for the second region being acquired from the drawing unit 11 subsequent to the first region. Run. Then, the display device output unit 12 synthesizes the second area subjected to the vibration correction process and the first area acquired from the drawing unit 11 in this order, and outputs the result to the display device 2 as the display image 700 .

FIG. 8 is a block diagram showing an example of the detailed configuration of the display device output section 12 according to this embodiment.

In this embodiment, the display device output unit 12 includes a dividing unit 121, a vibration correcting unit 122, a synthesizing unit 123, and a distortion correcting unit 124, as shown in FIG.

The dividing unit 121 divides the drawn image 701 formed by the drawing unit 11 and outputs the divided images to the vibration correcting unit 122 or the synthesizing unit 123 . More specifically, among the signals output from the drawing unit 11 , the dividing unit 121 outputs a signal indicating the second region to the vibration correcting unit 122 and outputs a signal indicating the first region to the synthesizing unit 123 . . In other words, the dividing unit 121 outputs the signal indicating the ST region to the vibration correcting unit 122 when acquiring the signal indicating the ST region from the drawing unit 11, and outputs the signal indicating the AR region to the vibration correcting unit 122 when acquiring the signal indicating the AR region. , outputs a signal indicating the AR region to the synthesizing unit 123 . In the present embodiment, when obtaining the signal indicating the drawn image 701, the dividing unit 121 first obtains the signal indicating the ST region and then obtains the signal indicating the AR region.

The vibration correction unit 122 performs vibration correction processing on the image output from the dividing unit 121 . More specifically, the vibration correction unit 122 executes vibration correction processing for correcting the display position of the virtual image of the second area based on the vibration information. More specifically, first, while the division unit 121 acquires a signal indicating the ST region from the rendering unit 11 and outputs the signal to the synthesis unit 123, the vibration correction unit 122 indicates the vibration amount of the moving body 100. Vibration information is acquired from the vibration detection unit 4 . Subsequently, while the signal indicating the AR region is being acquired from the dividing unit 121, the vibration correction unit 122 corrects the display position of the virtual image of the AR region with respect to the signal indicating the AR region based on the vibration information. Execute correction processing.

In other words, based on the vibration information acquired while the dividing unit 121 is outputting the signal indicating the ST region to the synthesizing unit 123, the vibration correcting unit 122 combines the actual scene due to the vibration of the moving body 100 with the output from the dividing unit 121. Vibration correction processing is performed to correct the display position of the virtual image in the AR area output from the dividing unit 121 so as to cancel the deviation of the virtual image in the AR area.

In this way, the vibration correction unit 122 executes the vibration correction process at a stage after the drawing unit 11 and closer to the display device 2 . Further, the vibration correction unit 122 acquires the signal indicating the ST region by the dividing unit 121, and uses the vibration information obtained during the period when the signal indicating the ST region is output to the combining unit 123 to correct the vibration of the signal indicating the AR region. Processing can be performed. With such vibration correction processing, even if the posture of the moving body 100 changes due to vibration, the virtual image 300 is placed at a specific position where it should be superimposed on the actual scene viewed from the user 200, for example, in the target space 400 shown in FIG. Is displayed.

Here, a detailed example of vibration correction processing will be described with reference to FIG.

FIG. 9 is a diagram for explaining a detailed example of vibration correction processing according to the present embodiment. Note that in FIG. 9, the reference numerals of the AR area image 702, which is an image indicating the AR area in the drawing image 701 before the vibration correction process is executed, and the arrow part 710 in the AR area image 702, are denoted by X is attached. Also, Y is attached to the reference numerals of the AR area image 702 and the arrow portion 710 after the vibration correction processing is executed.

The vibration correction unit 122 performs vibration correction processing on the AR area image 702 output from the division unit 121 to correct the deviation between the real scene and the virtual image caused by the posture change due to the vibration of the moving body 100 . The vibration correction unit 122 executes vibration correction processing using, for example, a buffer 1220 shown in FIG.

A buffer 1220 temporarily stores the AR area image 702 output from the dividing unit 121 . The buffer 1220 can change the reading start position P1 when reading the AR area image 702 from the buffer 1220 . For example, when the reading start position P1 is set to the upper left corner (reference position) of the AR area image 702X, one screen of image is read from the buffer 1220 with the upper left corner of the AR area image 702X as a reference. In other words, the AR area image 702X is read out from the buffer 1220 while the composition is not shifted.

On the other hand, when the reading start position P1 is changed to a position P1a that is shifted above the upper left corner of the AR area image 702X, an image for one screen is read from the buffer 1220 with reference to the reading start position P1a. . In this case, the AR area image 702X is read from the buffer 1220 with the composition shifted downward by the amount that the read start position P1a is shifted upward. As a result, when the AR area image 702Y is displayed on the display surface 20a, the arrow portion 710 in the AR area image 702Y is displayed shifted downward. Conversely, if the reading start position P1 is set to a position below the upper left corner of the AR area image 702, when the AR area image 702 is displayed on the display surface 20a, the arrow in the AR area image 702Y A portion 710Y is displayed shifted upward.

The vibration correction unit 122 detects the pitch angle of the posture change due to the vibration of the moving body 100 based on the vibration information acquired from the vibration detection unit 4, and cancels the deviation between the virtual image 300 and the real scene due to the pitch angle. , to change the read start position P1. For example, in the example shown in FIG. 3, when the pitch angle causes the moving body 100 to tilt backward, the virtual image 300a corresponding to the AR area image 702 is displayed below the default position 401 in the target space 400. As shown, the readout start position P1 is changed to a position P1a above the upper left corner of the AR area image 702 . In this way, the vibration correction process is executed by reading the AR area image 702 from the reading start position P1a. That is, when the AR area image 702Y after correction is displayed on the display device 2, the display position of the arrow portion 710Y in the AR area image 702Y is higher than the display position of the arrow portion 710X in the AR area image 702X before correction. corrected downwards. In this way, the deviation between the virtual image 300 and the real scene caused by the vibration of the moving body 100 is corrected.

If the reading start position P1 is changed to a position shifted from the upper left corner of the AR area image 702, the composition of the AR area image 702 is shifted, and a portion 702s having no image is generated in the AR area image 702. The no-image portion 702s may be displayed as a black image, for example.

In this way, the vibration correction process is performed after the drawing unit 11 and before the display device 2 . Therefore, display delay in the display device 2 can be suppressed.

In the above description, the posture change due to the vibration of the moving body 100 is the change in the pitch direction of the moving body 100, but it may be the change in the yaw direction of the moving body 100. FIG. In this case, the readout start position P1 is shifted left and right according to the yaw angle of the moving body 100. FIG. When the AR area image 702 is displayed on the display device 2, the display position of the arrow portion 710 in the AR area image 702 is shifted in the horizontal direction from the arrow portion 710 in the AR area image 702 before correction. corrected. Further, the posture change due to the vibration of the moving body 100 may be a change in the roll direction. In this case, the AR area image 702 is read so as to be rotated according to the roll angle with respect to the read start position P1. Then, when the AR area image 702 is displayed on the display device 2, the arrow portion 710 in the AR area image 702 is corrected to a position rotated with respect to the arrow portion 710 in the AR area image 702 before correction. The yaw direction is the direction around the vertical axis of the moving body 100, and the yaw angle is the rotation angle in the yaw direction. The roll direction is the angle around the longitudinal axis of the moving body 100, and the roll angle is the rotation angle in the roll direction.

The synthesizing unit 123 synthesizes the image output from the drawing unit 11 and the image on which vibration correction processing has been performed by the vibration correcting unit 122 . More specifically, the synthesizing unit 123 synthesizes the second area on which the vibration correction processing has been performed and the first area obtained from the drawing unit 11 in this order. In other words, the synthesizing unit 123 synthesizes, in this order, the signal indicating the AR region subjected to the vibration correction processing in the vibration correcting unit 122 and the signal indicating the ST region acquired from the dividing unit 121 to form the display image 700. do.

Here, a detailed example of the synthesizing process will be described with reference to FIG. 10 .

FIG. 10 is a diagram for explaining a detailed example of the combining processing according to this embodiment. For reference, FIG. 10A shows an AR area image 702X immediately before the vibration correction process is performed and an arrow portion 710X in the AR area image 702X. FIG. 10(b) shows an AR area image 702Y and an arrow portion 710Y after the vibration correction process is executed, that is, immediately before the synthesis process is executed. FIG. 10(c) shows an ST region image 703, which is an image representing the ST region output from the dividing unit 121 and before the synthesis processing is performed, and a velocity portion 711 in the ST region image 703. .

The synthesizing unit 123 combines, in this order, the AR area image 702Y shown in FIG. Synthesize. As a result, the synthesizing unit 123 forms a display image 700 shown in (d) of FIG.

Further, the synthesizing unit 123 may perform color tone correction processing for adjusting the color of each portion of the display image 700 on the formed display image 700 . More specifically, the synthesizing unit 123 may correct the color of each portion of the formed display image 700 so as to have a predetermined color. For example, if a portion defined as white in the display image 700 is reddish, the synthesizing unit 123 corrects the reddish portion to white. The synthesizing unit 123 then outputs the color-tone-corrected display image 700 to the distortion correcting unit 124 .

Note that the synthesizing unit 123 has a buffer (not shown) that temporarily stores the image output from the drawing unit 11 and the image on which vibration correction processing has been performed by the vibration correcting unit 122 for synthesizing processing. is doing. In this case, the synthesizing unit 123 may temporarily store the color tone-corrected display image 700 in the buffer and output it to the distortion correcting unit 124 .

The distortion correction unit 124 performs distortion correction processing on the display image 700 that has undergone color tone correction in the composition unit 123 . The distortion correction unit 124 outputs the display image 700 subjected to the distortion correction process to the display device 2 .

More specifically, the distortion correction unit 124 applies distortion correction processing different from the vibration correction processing to an image obtained by synthesizing the second region on which the vibration correction processing has been performed and the first region acquired from the drawing unit 11 in this order. is further executed and output to the display device 2 as a display image 700 . That is, the distortion correction unit 124 performs distortion correction processing different from the vibration correction processing on the output image synthesized by the synthesis unit 123 , and outputs the image as the display image 700 to the display device 2 .

The distortion correcting unit 124 corrects the display image 700 output by the synthesizing unit 123 so that the virtual image 300 is not distorted due to reflection on the windshield 101 and the projection unit 3, for example. Then, the distortion correction unit 124 outputs the display image 700 subjected to the distortion correction process to the display device 2 .

Note that the distortion correction unit 124 may perform distortion correction processing for each display image 700 color-tone-corrected by the synthesis unit 123, but the present invention is not limited to this. The distortion correction unit 124 may have a line buffer, and may read a small area image, which is an image area corresponding to several lines in the display image 700, from the line buffer to perform distortion correction processing. In this case, the distortion correction unit 124 outputs the image of each line of the small region image to the display device 2 after the distortion correction processing. As a result, the delay in image output from the line buffer to the display device 2 can be further suppressed, and as a result, the display delay on the display device 2 can be further suppressed.

Moreover, the drawing unit 11 and the display device output unit 12 may each be configured by a microcomputer having a CPU and a memory as main components. In this case, the drawing unit 11 and the display device output unit 12 are implemented by a computer having a CPU and a memory, respectively. It functions as the device output section 12 . The program for the drawing unit 11 is pre-recorded in the memory of the drawing unit 11, and the program for the display device output unit 12 is pre-recorded in the memory of the display device output unit 12. However, the program can be downloaded via an electric communication line such as the Internet or via a memory card. It may be provided by being recorded on a recording medium such as.

[Vibration detector 4]
The vibration detector 4 detects the amount of vibration of the moving body 100 and outputs it as vibration information. The vibration detection unit 4 is a sensor that detects vibration applied to the moving body 100, and is, for example, a gyro sensor (angular velocity sensor), but is not limited to this, and may be, for example, an acceleration sensor. The vibration detection unit 4 detects the pitch angle of the moving body 100 that changes due to the vibration of the moving body 100, for example, as information on the vibration applied to the moving body 100. FIG.

Note that the vibration detection unit 4 may be provided in the main body 1 and detect the vibration applied to the moving body 100 by detecting the vibration applied to the main body 1 . This is because the main body 1 is fixed to the moving body 100 , and therefore the vibration applied to the main body 1 is the same as the vibration applied to the moving body 100 .

(3) Operation Operation processing executed by the video display system 10 configured as described above will be described.

FIG. 11 is a flow chart showing operation processing executed by video display system 10 according to the present embodiment. FIG. 12 is a diagram showing an example of video formats and timings output by the drawing unit 11 and the display device output unit 12 according to this embodiment. Note that FIG. 12A shows the video format of the drawn image 701 output by the drawing unit 11 . (b) of FIG. 12 shows the video format of the display image 700 output by the display device output unit 12 .

First, in the video display system 10, the drawing unit 11 draws an image consisting of the ST area and the AR area, and outputs a signal indicating the ST area and a signal indicating the AR area in this order (step S10). More specifically, according to the video format shown in FIG. 702X) are output to the display device output unit 12 in this order.

Next, the display device output unit 12 acquires vibration information indicating the vibration amount of the moving body 100 during the first period in which the signal indicating the ST region is being obtained. Further, the display device output unit 12 performs vibration correction processing for correcting the display position of the virtual image in the AR region based on the vibration information for the AR region being acquired in the second period following the first period, The AR area on which the vibration correction process has been performed and the ST area obtained in the first period are synthesized in this order, and output as a display image to the display device 2 (step S20).

More specifically, as shown in FIG. 12, the display device output unit 12 acquires vibration information indicating the amount of vibration of the moving body 100 while the drawing unit 11 is outputting a signal indicating the ST region. . Subsequently, the display device output unit 12 calculates the correction amount (vibration correction amount) from the acquired vibration information, and the drawing unit 11 outputs the signal ST region indicating the ST region, followed by the AR region. Perform vibration correction processing on the signal. In other words, the display device output unit 12 outputs the signal indicating the AR region and the period during which the signal indicating the ST region is output by the drawing unit 11 at the stage after the drawing unit 11 and before the display device 2. The amount of correction is determined by the time the vibration starts, and vibration correction processing is performed on the signal indicating the AR region output from the drawing unit 11 .

After executing vibration correction processing, etc., the display device output unit 12 outputs a signal indicating the AR area (AR area image 702Y) and a signal indicating the ST area as the display image 700 according to the video format shown in FIG. (ST area image 703) are output to the display device 2 in this order.

In this way, the image display system 10 can reduce the delay from acquisition of vibration information to execution of correction processing for correcting the display position of the virtual image in the AR area. For this reason, the image display system 10 results in more vibration information than when the drawing unit 11 outputs the signal indicating the AR region and the signal indicating the ST region to the display device output unit 12 in this order. It is possible to quickly determine the correction amount and execute the vibration correction process. Therefore, the video display system 10 can further reduce the display delay of the image corresponding to the virtual image.

(4) Effect, etc. The effect of this embodiment will be described using a comparative example.

FIG. 13 is a diagram showing a comparative example, and FIG. 14 is an explanatory diagram of the effects of this embodiment. Note that (a) of FIG. 13 and (a) of FIG. 14 show the video format output by the drawing unit 11 . (b) of FIG. 13 and (b) of FIG. 14 show video formats output by the display device output unit 12. FIG. Further, as shown in FIGS. 13B and 14B, the display device output unit 12 performs vibration correction by shifting the readout start position around the time when the delay amount has passed after the time when the vibration information is acquired. A display image 700 that has been processed is output. Further, the delay amounts shown in FIGS. 13 and 14 include hardware delay amounts.

As shown in (a) of FIG. 13, in the comparative example, the video format output by the rendering unit 11 is reversed with respect to the present embodiment shown in (a) of FIG. That is, the drawing unit 11 according to the comparative example forms and outputs the output image 700Z, that is, the AR area and the ST area in this order in time series. In addition, in the HUD device 10A, a display image in which the AR area is positioned at the top and the ST area is positioned at the bottom is often displayed. output in the format On the other hand, in the present embodiment, the drawing unit 11 is caused to output a video format in which the ST area and the AR area are in this order.

Further, both the display device output unit 12 according to the comparative example and the present embodiment execute vibration correction processing using vibration information acquired immediately before the AR region is input. The display device output unit 12 according to the comparative example acquires the vibration information immediately before the signal indicating the AR region is input, for example, at time t0 before the output image 700Z is input from the drawing unit 11 . Further, the display device output unit 12 according to the comparative example determines the correction amount (vibration correction amount) of the AR area, performs the vibration correction process of the AR area, and outputs the AR area in which the vibration correction process has been performed to, for example, the time Output to the display device 2 from t4.

Therefore, in the comparative example, a relatively large amount of delay occurs from the acquisition of the vibration information to the execution of the vibration correction process and the output of the display image 700 to the display device 2 . In other words, in the comparative example, there is a performance constraint on content that requires real-time performance.

On the other hand, as shown in (a) of FIG. 14, the video format output by the rendering unit 11 according to the present embodiment is the opposite of the video format according to the comparative example, and the rendered image 701 is the ST region and the AR region. regions are output in this order in chronological order. The display device output unit 12 according to the present embodiment receives the signal indicating the ST region during the period in which the drawn image 701 is being input from the drawing unit 11 immediately before the signal indicating the AR region is input. Vibration information is acquired, for example, at time t2 during the period. Then, the display device output unit 12 according to the present embodiment determines the correction amount (vibration correction amount) of the AR area, performs the vibration correction process of the AR area, and outputs the AR area on which the vibration correction process has been executed. For example, it is output to the display device 2 from time t3.

As described above, the display device output unit 12 according to the present embodiment outputs a signal indicating the AR region and a signal indicating the ST region from the drawing unit 11 in this order, compared to the comparative example. The correction amount used for executing correction processing for correcting the display position of the virtual image can be determined more quickly from the vibration information. Therefore, the image display system according to the present embodiment can reduce the delay from acquisition of vibration information to execution of correction processing for correcting the display position of the virtual image in the second region. The image display delay can be further reduced.

(5) Modifications The embodiment described above is merely one of various embodiments of the present disclosure. The above-described embodiment can be modified in various ways according to design and the like, as long as the object of the present disclosure can be achieved. Furthermore, the aspect according to the above-described embodiment is not limited to being embodied in the video display system 10 as a single unit. For example, the mobile body 100 including the image display system 10 and the image display method using the image display system 10 may embody the aspect according to the above embodiment. Further, the mode according to the above-described embodiment may be embodied by a program for causing a computer to execute the above-described image display method, a recording medium storing this program, and the like. Modifications described below can be applied in combination as appropriate.

For example, the projection unit 3 only needs to have at least an optical element, it is not essential to have two mirrors, the first mirror 31 and the second mirror 32, and it has only one mirror or three or more mirrors. You may have Furthermore, the projection unit 3 may have optical components other than mirrors, such as lenses.

Further, the HUD device 10A is not limited to a configuration in which the virtual image 300 is projected onto the target space 400 set in front of the traveling direction of the moving body 100. You may project the virtual image 300 to .

Moreover, the video display system 10 is not limited to the HUD device 10A or the mobile object 100, and may be used in an amusement facility, for example. The video display system 10 may also be used in wearable terminals such as a head mounted display (HMD), medical equipment, or stationary devices. Further, the video display system 10 may be used as an electronic viewfinder or the like by being incorporated in a device such as a digital camera.

Further, in the present embodiment, the rendering unit 11 and the display device output unit 12 are configured by mutually different CPUs and memories. 12 may be shared.

Also, the drawing unit 11 may be configured with a GPU (Graphics Processing Unit) instead of being configured with a CPU. Also, the display device output unit 12 may be configured with an FPGA (field-programmable gate array) instead of the CPU.

Further, in the above-described embodiment, the display device 20 is configured to include the liquid crystal panel 201, but is not limited to such a configuration. The display 20 may be configured to scan the display surface 20 a of the display 20 from behind with a laser beam to form the display image 700 .

INDUSTRIAL APPLICABILITY The present disclosure can be used for a video display system, a video display method, and a program used for a HUD device that requires real-time performance in correction processing of the display position of the virtual image 300, and a moving body equipped with such a HUD device.

Reference Signs List 1 main unit 2 display device 3 projection unit 4 vibration detection unit 10 video display system 10A HUD device 11 drawing unit 12 display device output unit 20 display device 20a display surface 21 display control unit 31 first mirror 32 second mirror 100 moving body 101 Windshield 102 Dashboard 111 Rendering unit body 112 Rendering buffer 121 Divider 122 Vibration correction unit 123 Synthesis unit 124 Distortion correction unit 200 User 201 Liquid crystal panel 202 Light source device 300, 300a, 300b Virtual image 400 Object space 500 Optical axis 501 Virtual surface 600 Road surface 700 Display image 701 Drawing image 702, 702X, 702Y AR area image 702s Part 703 ST area image 710, 710X, 710Y Arrow part 711 Speed part 1220 Buffer

Claims (6)

A video display system in which a virtual image of a display image displayed by a display device is superimposed on a real scene in front of a moving body so as to be visible to a viewer,
An image consisting of a first area that is an area that does not move with time and a second area that is an area that moves with time is drawn, and a signal indicating the first area and a signal indicating the second area are generated in this order. a drawing unit to output;
Vibration information indicating the amount of vibration of the moving object is acquired while the first region is being acquired from the rendering unit, and the second region being acquired from the rendering unit is processed based on the vibration information. , performing vibration correction processing for correcting the display position of the virtual image of the second region, and synthesizing the second region subjected to the vibration correction processing and the first region acquired from the drawing unit in this order. and a display device output unit for outputting to the display device as the display image,
video display system. The display device output unit applies distortion correction processing different from the vibration correction processing to an image obtained by synthesizing the second region subjected to the vibration correction processing and the first region acquired from the drawing unit in this order. 2. The video display system according to claim 1, further executing and outputting to said display device as said display image. The display device output unit
a vibration correction unit that executes the vibration correction process based on the vibration information;
a synthesizing unit that synthesizes the second region on which the vibration correction processing has been performed and the first region obtained from the drawing unit in this order;
a dividing unit that outputs a signal indicating the second region to the vibration correcting unit and outputs a signal indicating the first region to the synthesizing unit, among the signals output from the drawing unit;
a distortion correction unit that performs distortion correction processing different from the vibration correction processing on the image synthesized by the synthesis unit and outputs the display image to the display device;
3. The image display system according to claim 2. Equipped with the video display system according to any one of claims 1 to 3,
Mobile. A video display method using a video display system in which a virtual image of a display image displayed by a display device is superimposed on a real scene in front of a moving body and visible to a viewer,
An image consisting of a first area that is an area that does not move with time and a second area that is an area that moves with time is drawn, and a signal indicating the first area and a signal indicating the second area are generated in this order. a drawing step to output;
Vibration information indicating the amount of vibration of the moving body is obtained during a period in which the signal indicating the first region is being obtained, and the vibration information is obtained for the second region being obtained in a period following the period. Based on this, vibration correction processing for correcting the display position of the virtual image of the second region is performed, and the second region subjected to the vibration correction processing and the first region acquired during the period are synthesized in this order. and an output step of outputting to the display device as the display image,
Image display method. to the computer,
A program for executing the video display method according to claim 5.

Images