JP7180702B2 - Display device, display system, and program - Google Patents

Description

The present invention relates to display devices, display systems, and programs.

2. Description of the Related Art Conventionally, a head-up display (HUD, Head-Up Display) that displays information directly in the field of view of a user is known in mobile devices such as automobiles and aircraft.

Since HUDs allow users (drivers) to visually recognize information without moving their line of sight, they are expected to reduce the physical burden associated with driving and prevent accidents caused by looking away. starting.

The HUD uses a light source such as a laser or LCD (Liquid Crystal Display) and projects it onto a windshield (windshield) or a combiner to display a virtual image that appears to be floating in space in front of the user's line of sight.

Consider a case where a HUD is used for augmented reality (AR) display, in which an object indicating information about an object such as an oncoming vehicle or navigation information such as route guidance is superimposed on the background of the real world. .

In this case, in the conventional technology, the object moves out of the area that can be displayed by the HUD as the object moves and the device on which the HUD is mounted moves. In this case, since the display of the object is missing at the edge (boundary) of the area that can be displayed by the HUD, the user may recognize the edge (boundary) of the display area of the HUD, resulting in a mismatch between the AR display and the background real world. There is a problem that the physical sensation fades and the reality of the AR display fades.

Therefore, it is an object of the present invention to provide a technique for making it difficult to recognize the boundary of a display area in which an object is superimposed on the background.

a display unit that displays an object superimposed on a background; a position of the object and a moving direction of the object; estimating the time or distance it takes for the object to reach the edge of an area displayable by a display unit, and adjusting the display mode of the object based on the estimated time or distance for the object to reach the edge of the area; and a computing device for determining, when the object enters the displayable region, the computing device gradually, at a constant speed, from the time the object enters the display region, from a display mode that is difficult for the user to visually recognize. and the display mode of the object at a second display position closer to the boundary of the displayable area of the display unit than at the first display position is changed to the display mode of the object at the first display position. is determined to be a display mode that is less visible .

According to the disclosed technique, it is possible to make it difficult to recognize the boundary of the display area in which the object is superimposed on the background.

It is a figure which shows an example of the system configuration|structure of the display system which concerns on embodiment. It is a figure explaining the example of installation of the display apparatus which concerns on embodiment. It is a figure explaining the display area of the display apparatus which concerns on embodiment. It is a figure showing an example of hardware constitutions of a display concerning an embodiment. It is a figure which shows an example of the functional block of the display apparatus which concerns on embodiment. It is a figure explaining AR display displayed by a display part. 9 is a flowchart showing an example of processing for determining the display mode of a target object according to the time required for the target object to move to the position of the edge of the display area; FIG. 8 is a diagram illustrating an example of processing for determining the display mode of a target object according to the time required for the target object to move to the edge position of the display area; It is a figure explaining the change of the display mode of a target object. 10 is a flowchart showing an example of processing for determining the display mode of a target object according to the distance in the display area between the current display position of the target object and the edge of the display area to which the target object is to be moved; FIG. 10 is a diagram illustrating processing when an object enters the display area; 9 is a flow chart showing an example of processing when an object enters the display area again after leaving the display area; FIG. 10 is a diagram illustrating an example of a display mode of an object when the object reenters the display area after exiting the display area; FIG. 10 is a diagram illustrating an example of a display mode when an object enters or exits the display area due to disturbance; FIG. 10 is a diagram illustrating an example of a display mode in which an object is rotated and displayed in a smaller size;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<System configuration>
First, the system configuration of the display system 1 according to this embodiment will be described using FIG. FIG. 1 is a diagram showing an example of a system configuration of a display system 1 according to an embodiment.

As shown in FIG. 1, a display system 1 according to the present embodiment is mounted on a mobile device such as a vehicle, an aircraft, or a ship, and includes a display device 10, one or more ECUs (Electric Control Units) 20-1, 20- 2, . . . Although an example in which the display system 1 is mounted on a vehicle will be described below, the display system 1 can also be applied to mobile devices other than vehicles.

The display device 10 and the ECU 20 are connected by a network such as CAN (Control Area Network).

The display device 10 is, for example, a HUD, and based on the information acquired from the ECU 20, displays an object indicating information about an object such as an oncoming vehicle, directions for turning right and left, and navigation information for route guidance such as intersection names. , superimposed on the background real world. Note that displaying an object overlaid on the background real world (on the background) is also called AR display.

FIG. 2 is a diagram illustrating an installation example of the display device 10 according to the embodiment. As shown in FIG. 2, the display device 10 is installed, for example, in a dashboard 503 in front of a driver 502 facing a steering wheel 501 (in the traveling direction of the mobile device 100). The display device 10 irradiates projection light 504 from, for example, an opening (projection opening) in a dashboard 503 and reflects it off a windshield 505 to show a virtual image to a driver 502, thereby displaying an object in a display area 506. do.

FIG. 3 is a diagram illustrating the display area of the display device 10 according to the embodiment. As shown in FIG. 3 , in the display area 506 of the display device 10 , an object 510 displayed by the display device 10 is displayed overlapping the real world seen through the windshield 505 . The display device 10 moves the object 510 in the display area 506 in conjunction with the position of the object in the background with which the object 510 is associated. As a result, the driver recognizes as if the object 510 actually exists there.

The ECU 20 displays information such as the speed of the own vehicle (vehicle speed), the steering angle (steering angle), the inclination of the own vehicle such as pitch and roll, the position and size of the object in real space, and the type of the object. to notify.

<Hardware configuration>
Next, the hardware configuration of the display device 10 according to this embodiment will be described using FIG. FIG. 4 is a diagram showing an example of the hardware configuration of the display device 10 according to the embodiment.

The display device 10 includes a communication I / F 101, a CPU (Central Processing Unit) 102, a ROM (Read Only Memory) 103, a RAM (Random Access Memory) 104, an imaging device 105, an external I / F 106, and an SSD (Solid State Drive ) 107 and the like. These pieces of hardware are connected to each other by bus B.

The communication I/F 101 is an interface that connects the display device 10 to a network such as CAN.

The CPU 102 reads programs and data from storage devices such as the ROM 103 and the SSD 107 onto the RAM 104 and executes processing based on the programs and data, thereby controlling the entire display device 10 and other functions. .

The ROM 103 is a non-volatile semiconductor memory that can retain programs and data even when power is turned off. The ROM 103 stores programs and data such as OS settings and network settings, and a BIOS (Basic Input/Output System) that is executed when the display device 10 is started. A RAM 104 is a volatile semiconductor memory that temporarily holds programs and data.

The imaging device 105 is a projection device that emits projection light from a laser, or a display such as an LCD that performs display using a light-emitting element, and displays the processing result of the display device 10 .

The SSD 107 is a nonvolatile storage device that stores programs and data. Programs and data stored in the SSD 107 include an OS (Operating System), which is basic software for controlling the entire display device 10, and application software that provides various functions on the OS. Also, the SSD 107 manages stored programs and data by a predetermined file system or DB.

The external I/F 106 is an interface with an external device. The external device includes a recording medium 106a and the like. The display device 10 can read from and write to the recording medium 106 a via the external I/F 106 . The recording medium 106a includes a flexible disk, CD, DVD, SD memory card, USB memory, and the like.

<Functional configuration>
Next, with reference to FIG. 5, the functional configuration of the display device 10 according to the embodiment will be described. FIG. 5 is a diagram showing an example of functional blocks of the display device 10 according to the embodiment.

The display device 10 has an acquisition unit 11 , an estimation unit 12 , a determination unit 13 and a display unit 14 . Each of these units is realized by processing that one or more programs installed in the display device 10 cause the CPU 102 of the display device 10 to execute.

The acquisition unit 11 acquires from the ECU 20 information such as the speed (vehicle speed) of the own vehicle, the steering angle of the steering wheel, the inclination of the own vehicle such as pitch and roll, the position and size of the object in real space, and the type of the object. The acquisition unit 11 also calculates and acquires the moving direction of the object in the real space based on the position of the object in the real space, the moving speed of the moving device, and the steering angle of the moving device.

Based on the position of the object in the real space acquired by the acquisition unit 11 and the moving direction of the object in the real space, the estimation unit 12 determines the area where the object can be displayed by the display unit 14 (hereinafter referred to as "display area"). ) to estimate the time to reach the edge of Alternatively, the estimation unit 12 estimates the pixel distance between the current display position of the object and the edge of the display area based on the position of the object and the moving direction of the object acquired by the acquisition unit 11 .

The determining unit 13 determines the display mode of the object based on the time or distance estimated by the estimating unit 12 .

The display unit 14 displays the object superimposed on the real world in the background of the display area seen from the user in the display mode determined by the determination unit 13 using projection light or light emitting elements.

FIG. 6 is a diagram for explaining the AR display displayed by the display unit 14. As shown in FIG. As shown in FIG. 6A, objects (AR objects) 510a and 510b are arranged in a virtual space whose scale matches that of the real world. As shown in FIGS. 6B and 6C, even when the mobile device or the object moves, the virtual image of the object in the display area 506 is displayed when viewed from the driver 502 riding the mobile device. is superimposed on the arranged position in the real world. For conversion between the coordinates in the real space (real world) and the coordinates in the display area 506, a known technique such as projective transformation that is generally used in 3D image processing or the like may be used.

In FIGS. 6(B) and 6(C), when the vehicle is going straight and approaching an intersection, arrow objects are placed on the road surface at regular intervals to guide the direction to turn at the next intersection. It is displayed in AR as if it were there. Since these arrow objects are arranged so as to be stationary in the real space, they are displayed moving closer to the own vehicle in synchronization with the movement of the ground accompanying the advance of the own vehicle. In this case, these arrow objects are displayed so as to move from the top to the bottom of the display area of the display device 10 as the host vehicle moves forward, and finally exit the display area.

<Processing>
<<Processing when an object exits the display area>>
Next, processing of the display device 10 according to the embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing an example of processing for determining the display mode of the target object according to the time required for the target object to move to the position of the edge of the display area. The following processing may be performed, for example, each time the display device 10 displays each display frame. Further, in the following processing, when there are a plurality of objects to be displayed, for example, the processing after step S102 is executed for each object.

In step S<b>101 , the acquisition unit 11 of the display device 10 acquires user operation information from the ECU 20 . The user's operation information includes vehicle speed and steering angle information of the host vehicle.

Subsequently, the acquisition unit 11 of the display device 10 acquires information on an object to be processed (hereinafter referred to as a "target object") from the ECU 20 (step S102). The information of the object to be displayed includes, for example, the position and size of the object in real space.

Subsequently, the acquiring unit 11 of the display device 10 calculates the moving direction of the target object relative to the own vehicle in the real space (step S103).

Subsequently, the acquisition unit 11 of the display device 10 calculates the display position of the target object in the display area of the display device 10 (step S104).

Subsequently, the estimation unit 12 of the display device 10 estimates the moving direction of the target object in the display area of the display device 10 (step S105). For example, if the target object is stationary in the real space, such as an object that indicates information related to navigation such as route guidance, the target object over time is calculated based on the speed (vehicle speed) and steering angle of the own vehicle. can estimate the trajectory (direction of movement) of the display position of For example, if the target object is moving in real space, such as an object that indicates information about an object such as an oncoming vehicle, the relative speed between the object and the own vehicle measured by a stereo camera, radar, etc. The trajectory (moving direction) of the display position of the target object over time can be estimated based on the moving direction and the steering angle.

Subsequently, the estimation unit 12 of the display device 10 estimates the time required for the target object to move from the current display position to the edge position of the display area of the display device 10 (step S106). For example, based on the trajectory (moving direction) of the display position of the target object over time estimated in step S105, the time elapsed until the display position of the target object moves to the position of the edge of the display area. , the time until it moves to the position of the edge of the display area. Note that the display device 10 may estimate the time until the target object moves to the position of the edge of the display area, taking into account the information on the acceleration of the own vehicle.

Subsequently, the determination unit 13 of the display device 10 determines the display mode of the target object according to the time (step S107).

Subsequently, the display unit 14 of the display device 10 displays the target object in the determined display mode (step S108).

Next, the process of determining the display mode of the target object in step S107 will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of processing for determining the display mode of the target object according to the time required for the target object to move to the edge position of the display area.

If the target object 510 has a complicated shape as shown in FIG. 8, for example, a rectangle or the like that circumscribes the target object 510 may be used as the target object boundary 520 . Then, the time estimated to elapse until the boundary 520 of the target object 510 touches the edge of the display area 506 is calculated as the time until the target object 510 moves to the edge position 521 of the display area 506. good.

Next, changes in the display mode of the target object will be described with reference to FIG. FIG. 9 is a diagram illustrating changes in the display mode of the target object.

As shown in FIG. 9A, the determination unit 13 of the display device 10 determines that the normal display mode is changed as the estimated value of the required time until the boundary of the target object touches the edge of the display area decreases. From 550 , the display modes are gradually changed to display modes 551 to 554 that are less visible to the user. For example, when the estimated value of the time required to contact the edge of the display area becomes equal to or less than a predetermined value (for example, 4 seconds), the display device 10 reduces the estimated value as in the display modes 551 to 554. Increase the transparency of the target object along with . As a result, the target object becomes transparent as it approaches the edge of the display area, making it difficult for the user to notice the edge of the display area.

FIG. 9B is a diagram showing an example in which the target object slowly approaches the edge of the display area compared to FIG. 9A. As shown in FIG. 9B, when the target object relatively slowly approaches the edge of the display area, the target object gradually approaches the edge of the display area and then gradually becomes less visible to the user. be changed.

<Modification>
In FIG. 8, the example of determining the display mode of the target object according to the time until the target object moves to the position of the edge of the display area has been described. An example of determining the display mode of the target object according to the distance in the display area between the current display position of the target object and the edge of the display area to which the target object is to be moved will be described below.

FIG. 10 is a flowchart showing an example of processing for determining the display mode of the target object according to the distance in the display area between the current display position of the target object and the edge of the display area to which the target object is to be moved.

The processing of steps S201 through S205 and step S208 is the same as the processing of steps S101 through S105 and step S108 in FIG.

In step S206, the estimation unit 12 of the display device 10 estimates the distance in the display area between the current display position of the target object and the edge of the display area to which the target object is to be moved. For example, based on the trajectory (moving direction) of the display position of the target object over time estimated in step S205, display of the current display position of the target object and the edge of the display area to which the target object moves. A distance in the region (distance 523 shown in FIG. 8) can be calculated. Note that the distance in the display area corresponds to the distance in pixels when the display area is, for example, a liquid crystal display.

Subsequently, the determination unit 13 of the display device 10 determines the display mode of the target object according to the distance (step S207).

<<Processing when an object enters the display area>>
Next, with reference to FIG. 11, the processing of the determination unit 13 when an object enters the display area will be described. FIG. 11 is a diagram illustrating processing when an object enters the display area.

When an object enters the display area, for example, the determination unit 13 of the display device 10 gradually changes the display from display mode 554 to display mode 550 at a constant speed so that the display is difficult for the user to see. Mode 554 is changed to normal display mode 550 that is easily visible to the user. In this case, regardless of the moving speed and moving direction of the objects, the time required from entering the area to displaying the normal display mode 550 is the same for all objects.

<<Processing when an object reenters the display area after it exits the display area>>
Next, with reference to FIGS. 12 and 13, the processing of the determination unit 13 when the object enters the display area again after leaving the display area will be described. FIG. 12 is a flowchart showing an example of processing when an object reenters the display area after exiting the display area. FIG. 13 is a diagram for explaining an example of how an object is displayed when the object reenters the display area after exiting the display area.

In step S301, the determination unit 13 of the display device 10 detects that the object has entered the display area again after leaving the display area.

Subsequently, the acquisition unit 11 of the display device 10 acquires the type of object from the ECU 20 (step S302). The types of objects include, for example, information indicating vehicles, pedestrians, route guidance, and the like.

Subsequently, the determination unit 13 of the display device 10 determines whether or not to display the object again based on the type of the object acquired by the acquisition unit 11 (step S303). For example, in the case of an object of a preset type with a high degree of importance, it is determined that the object should be displayed again.

If it is not determined to display the object again (NO in step S303), the display unit 14 of the display device 10 does not display the object (step S304).

FIG. 13A is a diagram showing a display example when it is not determined to display again. In this case, the object that has entered the display area again is not displayed. As a result, objects that are not so important are not redisplayed, thereby preventing an increase in the driver's cognitive load.

If it is determined to display the object again (YES in step S303), the display unit 14 of the display device 10 displays the object in the determined display mode (step S305).

FIG. 13B is a diagram showing a display example when it is determined to display again. In this case, the object re-entering the display area is displayed in the same display mode as the above-described "process when object enters display area" and "process when object exits display area". This allows objects of relatively low importance to be seen by the driver for a longer period of time.

Note that, if the object is moved out of the display area by steering the moving device and then re-enters the display area by steering to cancel the steering, the determination unit 13 of the display device 10 displays the object on the display unit. 14 may be displayed. Thus, for example, when the driver turns the steering wheel to avoid an obstacle on the road and then turns the steering wheel in the opposite direction to return to the original traveling direction, the object can be redisplayed.

<<Processing when an object enters or exits the display area due to a disturbance>>
Next, with reference to FIG. 14, the processing of the determination unit 13 when an object enters or exits the display area due to disturbance will be described. FIG. 14 is a diagram illustrating an example of a display mode when an object enters or exits the display area due to disturbance.

In the above-mentioned "processing when an object exits the display area", an example of determining the display mode based on the vehicle speed and steering angle by accelerator operation and steering (steering wheel operation), which are vehicle operations by the driver, has been described. In addition, in the above-mentioned "processing when an object enters the display area", an example of determining the display mode according to the elapsed time after entering the display area has been described.

The display mode of an object when the object exits or enters the display area due to the inclination (disturbance) of the own vehicle, which is a factor other than the driver's vehicle operation, will be described below.

Disturbances include, for example, the bounce of the own vehicle due to discontinuous road shapes. If the display position of the object in the display area is not moved due to a sudden factor such as bounce, the display position of the object in the display area will deviate from the position of the object in the background in the real world, and the reality as AR will be lost. descend.

In order to prevent this, the display device 10 moves the object in synchronization with the background based on the pitch and roll of the own vehicle acquired from the ECU 20 . The pitch and roll of the own vehicle are acquired by an inclination sensor or the like installed in the own vehicle.

In this case, for example, if the vehicle bounces too much, the object may move out of the display area. In such a case, as shown in FIG. 14, the object is moved without changing the display mode depending on the disturbance. FIG. 14A shows an example in which the object is displayed in a normal display mode 550 before bouncing. FIG. 14B shows an example in which the object has left the display area while bounding. FIG. 14C shows an example in which the object is displayed in a normal display mode 550 after bouncing.

<Modified example of display mode>
In the above, an example of increasing the transparency has been described as a display mode that is difficult for the user to visually recognize. Alternatively, as a display mode that is less visible to the user, a display mode in which the object is displayed in a smaller size as a result of being rotated may be used.

FIG. 15 is a diagram illustrating an example of a display mode in which an object is displayed smaller by being rotated. FIG. 15 shows an example in which the object is rotated about the vertical direction. In this case, the determination unit 13 may determine the amount of rotation of the object, for example, in the display modes 551a to 554a by the same processing as the processing for determining the transparency described above. Note that the direction of rotation may be arbitrary. Also, the object may be rotated around the horizontal direction.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the invention described in the claims.・Changes are possible.

For example, the display device 10 and client device 30 may be configured as an integrated device.

Further, each functional unit of the display device 10, for example, the acquiring unit 11, the estimating unit 12, the determining unit 13, etc., may be realized by cloud computing configured by one or more computers.

1 display system 10 display device 11 acquisition unit 12 estimation unit 13 determination unit 14 display unit 20 ECU (an example of a “control device”)

JP 2012-242465 A

Claims (13)

a display unit for superimposing and displaying an object on a background;
obtaining the position of the object and the direction of movement of the object;
Based on the obtained position of the object and the moving direction of the object, the time until the object moves to the edge of the area displayable by the display unit, or the display position of the object and the movement destination of the object. estimating the distance to said edge of
a computing device that determines a display mode of the object based on the estimated time or distance;
When the object enters the displayable area, the arithmetic device gradually changes the display mode from a display mode that is difficult for the user to visually recognize to a display mode that is easy for the user to view at a constant speed from the time the object enters the display area. , the display mode of the object at a second display position closer to the boundary of the displayable area of the display unit than at the first display position, and the display mode of the object at the first display position; A display device, characterized in that a display mode that is difficult to be visually recognized is determined . 2. The display device according to claim 1 , wherein the hard-to-see display mode is a highly transparent display mode. 2. The display device according to claim 1 , wherein the difficult-to-visual display mode is a display mode in which the object is displayed in a smaller size by being rotated. 4. The display device according to any one of claims 1 to 3 , wherein the arithmetic device calculates the moving direction of the object based on the position of the object, the moving speed of the moving device, and the steering angle of the moving device. 5. The display device according to any one of claims 1 to 4 , wherein the computing device further estimates the time until the object moves to the edge based on the acceleration of the moving device. 6. The display device according to any one of claims 1 to 5 , wherein the arithmetic device does not change the display mode when the object moves in the area due to the tilt of the moving device. 7. The display device according to any one of claims 1 to 6 , wherein when the object enters the region after the object has left the region, the arithmetic device does not display the object on the display unit. . wherein the computing device determines, based on the type of the object, whether to display the object on the display unit when the object enters the area after the object has left the area; Item 8. The display device according to item 7 . wherein said arithmetic unit displays said object on said display unit when said object enters said area by steering to cancel said steering after said object has left said area by steering of said moving apparatus. 9. The display device according to 7 or 8 . 10. The display device according to any one of claims 1 to 9 , wherein the display unit displays the object overlaid on a background using projection light or a light emitting element. A moving object comprising the display device according to claim 1 . A display system comprising a display device having a display unit that displays an object overlaid on a background, and a control device that controls a moving device,
The control device is
Notifying the display device of data on the position of the object, the moving speed of the moving device, and the steering angle of the moving device;
The display device
obtaining the position of the object and the moving direction of the object calculated based on the data;
Based on the obtained position of the object and the moving direction of the object, the time until the object moves to the edge of the area displayable by the display unit, or the display position of the object and the movement destination of the object. estimating the distance to said edge of
a computing device that determines a display mode of the object based on the estimated time or distance;
When the object enters the displayable area, the arithmetic device gradually changes the display mode from a display mode that is difficult for the user to visually recognize to a display mode that is easy for the user to view at a constant speed from the time the object enters the display area. , the display mode of the object at a second display position closer to the boundary of the displayable area of the display unit than at the first display position, and the display mode of the object at the first display position; A display system characterized by determining a display mode that is difficult to be visually recognized . to the display device,
obtaining the position of an object and the direction of movement of said object;
A time until the object reaches the edge of an area that can be displayed by a display unit that displays the object overlaid on a background based on the obtained position of the object and the moving direction of the object, or estimating a distance between a display position of an object and the edge to which the object is moved;
determining a display mode of the object based on the estimated time or distance;
When the object enters the displayable area, the display mode is gradually changed at a constant speed from the time the object enters the display area to the display mode that is easily visible to the user , and the first display is performed. The display mode of the object at a second display position closer to the boundary of the displayable area of the display unit compared to the position, and the display mode of the object at the first display position is less visible than the display mode of the object. A program that decides to

Images