JP7351734B2 - display control device - Google Patents

Description

The present disclosure relates to a display control device.

Patent Document 1 discloses a vehicle display device. A vehicle display device displays a display object superimposed on an actual scene such as a road surface.

WO2018/030320A1

There may be a slope on the road in front of the vehicle. If the display position of the displayed object is constant, the distance between the displayed object and the road surface will change depending on the gradient. For example, suppose that the displayed object is displayed so as to be in contact with the road surface when there is no slope. If the display position of the display object is constant, if there is a downward slope in front of the vehicle, the display object will be displayed floating off the road surface. Furthermore, if the display position of the display object is constant, if there is an upward slope in front of the vehicle, the display object will appear as if it has sunk into the road surface.

In one aspect of the present disclosure, it is preferable to provide a display control device that can suppress a change in the apparent distance between a display object and the road surface even when there is a slope on the road in front of the vehicle.

One aspect of the present disclosure is a display control device (1) that uses a display device (47) to display a display object (71) superimposed on a scenery (73) in front of a vehicle (3). The display control device includes an image acquisition unit (9) configured to acquire an image (53) representing the scenery in front of the vehicle, and detects an upper end (61) of a road area (55) in the image. an upper end detection unit (11) configured to estimate the slope of the road; a slope estimation unit (13) configured to estimate the slope of the road; The display object is configured to be displayed at the display position determined by the display position determination unit using a display position determination unit (15) configured to determine a display position in the vertical direction and the display device. a display unit (17).

The slope estimation unit comprises a top edge usage unit (31) configured to estimate the slope of the road based on the position of the top edge in the image.
A display control device that is one aspect of the present disclosure estimates the gradient of a road. A display control device that is one aspect of the present disclosure determines a display position of a display object based on the estimated road slope. Therefore, according to the display control device that is one aspect of the present disclosure, even if the road has a slope, it is possible to suppress a change in the apparent distance between the displayed object and the road surface.

A display control device that is one aspect of the present disclosure estimates the gradient of a road based on the position of the upper end of the road area. Therefore, the display control device that is one aspect of the present disclosure can easily estimate the slope of the road. Furthermore, the display control device that is one aspect of the present disclosure can estimate the gradient of a road even when other estimation methods cannot be used.

FIG. 2 is a block diagram showing the configuration of a display control device. FIG. 2 is a block diagram showing the functional configuration of a display control device. 3 is a flowchart illustrating display processing executed by the display control device. 3 is a flowchart illustrating display processing executed by the display control device. 3 is a flowchart representing a road area detection process executed by the display control device. It is an explanatory view showing a template and a front image. FIG. 3 is an explanatory diagram showing a front image, a road area, and an upper end. FIG. 3 is an explanatory diagram illustrating a method of estimating a forward slope using the position of the upper end of a road area. It is an explanatory view showing an example of a display object.

Exemplary embodiments of the present disclosure will be described with reference to the drawings.
<First embodiment>
1. Configuration of Display Control Device 1 The configuration of the display control device 1 will be explained based on FIGS. 1 and 2. As shown in FIG. 1, the display control device 1 is mounted on a vehicle 3. The display control device 1 includes a microcomputer having a CPU 5 and a semiconductor memory (hereinafter referred to as a memory 7) such as a RAM or a ROM.

Each function of the display control device 1 is realized by the CPU 5 executing a program stored in a non-transitional physical recording medium. In this example, the memory 7 corresponds to a non-transitional physical recording medium that stores a program. Also, by executing this program, a method corresponding to the program is executed. Note that the display control device 1 may include one microcomputer or a plurality of microcomputers.

As shown in FIG. 2, the display control device 1 includes an image acquisition unit 9, an upper end detection unit 11, a gradient estimation unit 13, a display position determination unit 15, a display unit 17, and a lane boundary line detection unit 19. , a position information acquisition unit 21, a readout unit 23, a target object detection unit 25, a reliability calculation unit 27, and a clarity determination unit 29.

The gradient estimation unit 13 includes an upper end usage unit 31, a lane boundary usage unit 33, a map usage unit 35, and a target usage unit 37.
The display control device 1 is connected to various components included in the vehicle 3. As shown in FIG. 1, components connected to the display control device 1 include a camera 39, a GPS 41, a storage section 43, a radar 45, and a head-up display (HUD) 47.

The camera 39 generates an image representing the scenery in front of the vehicle 3 (hereinafter referred to as a front image 53). The camera 39 sends the front image 53 to the display control device 1. The GPS 41 acquires position information of the vehicle 3.
The storage unit 43 stores map information 49. The map information 49 includes slope information at multiple locations. Slope information is information representing the slope of a road. The display control device 1 can read the map information 49.

Radar 45 can detect a target existing in front of vehicle 3. Examples of targets include other vehicles, pedestrians, fixed objects, and the like. The radar 45 sends a signal representing the target object detection result to the display control device 1 . The signal representing the detection result of the target is a signal representing the distance from the vehicle 3 to the target, the direction of the target with respect to the vehicle 3, and the like. The orientation of the target includes an orientation in the horizontal direction and an orientation in the vertical direction.

The HUD 47 displays objects superimposed on the scenery in front of the vehicle 3. The HUD 47 corresponds to a display device. Examples of display objects include figures, characters, numbers, symbols, and the like. Examples of the figure include an arrow, a band-shaped figure, a straight line, a curved line, a polygon, and the like.

The display object is, for example, a display object (hereinafter referred to as a virtual display object) that appears as if it virtually exists in a three-dimensional space when viewed from the driver of the vehicle 3. The virtual display object is, for example, a display object that appears to be virtually at a constant distance from the road surface. The virtual display object is, for example, a display object that virtually extends along the road surface.

2. Process Executed by Display Control Device 1 The display process repeatedly executed by the display control device 1 at predetermined time intervals will be explained based on FIGS. 3 to 9. In step 1 of FIG. 3, the image acquisition unit 9 acquires the front image 53 using the camera 39.

In step 2, the upper end detection unit 11 detects a road area 55 in the forward image 53 acquired in step 1. The process of detecting the road area 55 will be explained based on FIGS. 5 and 6.
The upper end detection unit 11 includes N templates 51. N is a natural number of 1 or more. As shown in FIG. 6, the template 51 is smaller than the front image 53. Each of the N templates 51 is numbered from 1 to N. Each of the N templates 51 has a brightness similar to that of a general road. The brightness of the N templates 51 differs from template to template.

In step 31, the upper end detection unit 11 sets the value of i to 1. i is the number of the template 51.
In step 32, the upper end detection unit 11 selects the i-th template 51.

In step 33, the upper end detection unit 11 overlaps the i-th template 51 at the initial position in the front image 53. The initial position is a preset position. A portion of the front image 53 that overlaps with the template 51 is hereinafter referred to as a comparison area.

In step 34, the upper end detection unit 11 obtains the brightness of the comparison area.
In step 35, the upper end detection unit 11 determines whether the brightness of the template 51 and the brightness of the comparison area obtained in step 34 are close to each other. The fact that the brightness is close means that, for example, the difference in brightness is less than or equal to a preset threshold. If the brightness of the template 51 and the brightness of the comparison area are close, the process proceeds to step 36. If the brightness of the template 51 and the brightness of the comparison area are not close, the process proceeds to step 37.

In step 36, the upper end detection unit 11 determines that the comparison area is a road area.
In step 37, the upper end detection unit 11 determines that the comparison area is not a road area.
In step 38, the upper end detection unit 11 determines whether the position of the template 51 is the end position. The end position is the position of the template 51 when the template 51 has finished searching the entire area of the front image 53. If the position of the template 51 is not the end position, the process proceeds to step 39. If the position of the template 51 is the end position, the process proceeds to step 40.

In step 39, the upper end detection unit 11 moves the template 51 to the next position. After step 39, the process proceeds to step 34.
In step 40, the upper end detection unit 11 determines whether the value of i is N. Note that the value of i being N means that the processes of steps 32 to 39 have been performed for all templates 51. If the value of i is N, the process of detecting the road area ends and the process proceeds to step 3 in FIG. If the value of i is not N, the process proceeds to step 41.

In step 41, the upper end detection unit 11 increases the value of i by one. After step 41, the process proceeds to step 32.
As a result of the process of detecting a road area, a road area 55 is detected in the forward image 53, as shown in FIG. Note that when performing the process of detecting the road area 55, it is also possible to exclude in advance an area where a vehicle or structure is present from the forward image 53, and then detect the road area 55 in the area that has not been excluded. . An area where a vehicle or structure is located can be detected by a template matching method using a template of the vehicle or structure.

Returning to FIG. 3, in step 3, the lane boundary detection unit 19 detects lane boundaries in the forward image 53 acquired in step 1. Examples of lane boundary lines include white lines. A method for detecting lane boundary lines is, for example, as follows. The lane boundary detection unit 19 extracts edges in the forward image 53. An edge is a pixel whose brightness changes suddenly. The lane boundary line detection unit 19 detects lane boundary lines using a known method based on the extracted edges.

In step 4, the lane boundary line use unit 33 estimates the slope of the road in front of the vehicle 3 (hereinafter referred to as front slope θ) based on the lane boundary line detected in step 3. The forward slope θ is the slope of the road 59 in front of the vehicle 3 with reference to the road 57 on which the vehicle 3 exists, as shown in FIG.

There is a correlation between the direction in which the lane boundary line extends in the forward image 53 and the forward slope θ. The lane boundary line usage unit 33 estimates the forward slope θ based on the direction in which the lane boundary line extends in the forward image 53.

In step 5, the reliability calculation unit 27 calculates the reliability R1 of the forward slope θ estimated in step 4. The reliability calculation unit 27 calculates the reliability R1 higher as the number of edges extracted in step 3 increases.

In step 6, the reliability calculation unit 27 determines whether the reliability R1 calculated in step 5 is greater than or equal to a preset threshold. If the reliability R1 is greater than or equal to the threshold, the process proceeds to step 22 in FIG. If the reliability R1 is less than the threshold, the process proceeds to step 7.

In step 7, the upper end detection unit 11 detects the upper end 61 shown in FIG. 7 in the front image 53 acquired in step 1. The upper end 61 is the upper end of the road area 55.
When the upper end 61 is displayed in the front image 53, the upper end detection unit 11 directly detects the upper end 61. In the forward image 53, when the upper end 61 is hidden by a target or the like, the upper end detection unit 11 extends the left and right boundaries 63, 65 of the road area 55 forward. The upper end detection unit 11 determines the intersection of the boundary line 63 and the boundary line 65 as the upper end 61.

In step 8, the upper end use unit 31 estimates the forward slope θ based on the position of the upper end 61 detected in step 7. A method for estimating the forward gradient θ will be explained below. As shown in FIG. 8, the vanishing point when the forward slope θ is 0 degrees is defined as a reference vanishing point 67. The reference vanishing point 67 is located at a fixed position within the angle of view of the camera 39. θ′ shown in FIG. 8 is expressed by the following equation (1).

In equation (1), d is the distance between the upper end 61 and the reference vanishing point 67 at the imaging position 69 of the camera 39. F in equation (1) is the focal length of the camera 39. The upper end use unit 31 calculates θ' using equation (1). θ' approximates the forward slope θ. Therefore, the upper end use unit 31 uses the calculated θ' as the forward slope θ.

In step 9, the reliability calculation unit 27 calculates the reliability R2 of the forward slope θ estimated in step 8. The reliability calculation unit 27 calculates statistical values of brightness correlation values over the entire road area 55. The brightness correlation value is a value representing the closeness between the brightness of the template 51 and the brightness of the comparison area compared in step 35 above. The closer the brightness of the template 51 and the brightness of the comparison area are, the larger the correlation value becomes. The reliability calculation unit 27 calculates the reliability R2 higher as the statistical value is larger.

In step 10, the reliability calculation unit 27 determines whether the reliability R2 calculated in step 9 is greater than or equal to a preset threshold. If the reliability R2 is equal to or greater than the threshold, the process proceeds to step 22 in FIG. If the reliability R2 is less than the threshold, the process proceeds to step 11.

In step 11, the target object detection unit 25 executes a process of detecting a target object in front of the vehicle 3 using the radar 45.
In step 12, the target object detection unit 25 determines whether or not the target object was detected in the process of step 11. If the target object is detected, the process proceeds to step 13. If no target object is detected, the process proceeds to step 16.

In step 13, the target using unit 37 acquires the direction of the target detected in the process of step 11. The azimuth of the target is the azimuth with respect to the vehicle 3. There is a correlation between the direction of the target and the forward slope θ. For example, the forward slope θ is an uphill slope, and the larger the value of the forward slope θ, the higher the direction of the target is. The target using unit 37 estimates the forward slope θ based on the direction of the target.

When a plurality of targets are detected in the process of step 11, the target using unit 37 first estimates the forward gradient θ for each target. Next, the target using unit 37 estimates the final forward slope θ by statistically processing the forward slope θ for each target.

In step 14, the reliability calculation unit 27 calculates the reliability R3 of the forward gradient θ estimated in step 13. The reliability calculation unit 27 calculates the reliability R3 higher as the number of targets detected in the process of step 11 increases.

In step 15, the reliability calculation unit 27 determines whether the reliability R3 calculated in step 14 is greater than or equal to a preset threshold. If the reliability R3 is equal to or greater than the threshold, the process proceeds to step 22 in FIG. If the reliability R3 is less than the threshold, the process proceeds to step 16.

In step 16, the position information acquisition unit 21 uses the GPS 41 to acquire position information of the vehicle 3.
In step 17, the reading unit 23 determines whether the map information 49 can be read from the storage section 43. If the map information 49 can be read from the storage unit 43, the process proceeds to step 18 in FIG. If the map information 49 cannot be read from the storage unit 43, the process proceeds to step 21 in FIG.

In step 18, the reading unit 23 first reads the map information 49 from the storage section 43. Next, the map using unit 35 estimates the forward slope θ by comparing the position information of the vehicle 3 acquired in step 16 with the read map information 49.

In step 19, the reliability calculation unit 27 calculates the reliability R4 of the forward gradient θ estimated in step 18. The reliability calculation unit 27 calculates the reliability R4 higher as the reliability value of the position information of the vehicle 3 acquired in step 16 is higher. For example, the reliability value of the position information of the vehicle 3 increases as the number of GPS satellites whose signals can be received increases.

In step 20, the reliability calculation unit 27 determines whether the reliability R4 calculated in step 19 is greater than or equal to a preset threshold. If the reliability R4 is equal to or greater than the threshold, the process proceeds to step 22. If the reliability R4 is less than the threshold, the process proceeds to step 21.

In step 21, the slope estimating unit 13 sets the value of the forward slope θ to the value estimated in the previous display process.
In step 22, the display position determining unit 15 determines the vertical display position of the display object based on the forward slope θ. The forward gradient θ to be used is the forward gradient θ estimated in step 4 if the determination in step 6 is affirmative.

Further, the forward gradient θ to be used is the forward gradient θ estimated in step 8 if the determination in step 10 is affirmative. Further, the forward slope θ to be used is the front slope θ estimated in the step 13 if an affirmative determination is made in the step 15 above.

Further, the forward slope θ to be used is the front slope θ estimated in the step 18 if the determination in step 20 is affirmative. Furthermore, if a negative determination is made in step 20, the forward slope θ to be used is the front slope θ estimated in the previous display process.

The method by which the display position determining unit 15 determines the display position is as follows. The display position determining unit 15 defines a plane in three-dimensional space having a pitch angle corresponding to the forward slope θ. The display position determining unit 15 virtually arranges a display object on a plane in three-dimensional space. The display position determining unit 15 converts a plane and a display object in a three-dimensional space into an image in a two-dimensional space. The display position determining unit 15 employs the position of the display object in the converted image.

In step 23, the clarity determination unit 29 determines the clarity of the boundary between the displayed object and its surroundings according to the reliability. The reliability used is the reliability of the forward slope θ used in the process of step 22. For example, the clarity determination unit 29 increases the clarity of the boundary as the confidence level increases. Examples of modes in which the clarity of the boundary is low include modes in which the boundary is gradated, modes in which part or all of the displayed object is highly transparent, and the like.

In step 24, the display unit 17 uses the HUD 47 to display the display object. The display position is the position determined in step 22 above.
FIG. 9 shows an example of the display object 71. The display object 71 is superimposed on the scenery 73 in front of the vehicle 3 when viewed from the viewpoint of the driver of the vehicle 3. When viewed from the driver, the display object 71 appears as if it virtually exists in a three-dimensional space. The display object 71 extends along the road surface of the road 59 ahead. The display object 71 is a display object in which the distance from the road surface of the road 59 ahead appears to be constant regardless of the presence or absence of the forward slope θ and the magnitude of the forward slope θ.

3. Effects of the display control device 1 (1A) The display control device 1 estimates the forward slope θ. The display control device 1 determines the display position of the display object based on the estimated forward slope θ. Therefore, according to the display control device 1, even if there is a forward slope θ, the displayed object can be displayed so that the distance between the displayed object and the road surface appears constant.

The display control device 1 estimates the forward slope θ based on the position of the upper end 61 in the forward image 53. Therefore, the display control device 1 can easily estimate the forward slope θ. Furthermore, the display control device 1 can estimate the forward slope θ even when other estimation methods cannot be used.

(1B) The display control device 1 detects lane boundary lines in the forward image 53. The display control device 1 estimates the forward slope θ based on the direction in which the lane boundary line extends in the forward image 53. Therefore, the display control device 1 can estimate the forward slope θ more accurately. Furthermore, the display control device 1 can estimate the forward slope θ even when other estimation methods cannot be used.

(1C) The display control device 1 acquires position information of the vehicle 3. Display control device 1 reads map information 49 from storage unit 43 . The display control device 1 estimates the forward slope θ by comparing the position information of the vehicle 3 and the map information 49. Therefore, the display control device 1 can estimate the forward slope θ more accurately. Furthermore, the display control device 1 can estimate the forward slope θ even when other estimation methods cannot be used.

(1D) The display control device 1 detects a target existing in front of the vehicle 3 using the radar 45. The display control device 1 estimates the forward slope θ based on the direction of the detected target. Therefore, the display control device 1 can estimate the forward slope θ more accurately. Furthermore, the display control device 1 can estimate the forward slope θ even when other estimation methods cannot be used.

(1E) The display control device 1 calculates the reliability of the estimated forward slope θ. The display control device 1 determines the clarity of the boundary between the displayed object and its surroundings according to the reliability. Therefore, when the reliability is low, the clarity of the boundary of the displayed object is low. As a result, even if there is a difference in the distance between the displayed object and the road surface, the difference becomes less noticeable.
<Other embodiments>
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments and can be implemented with various modifications.

(1) The display device that displays the display object may be a display device other than the HUD 47. For example, it may be a display device in which a display element is embedded in a windshield.
(2) The method of detecting the road area 55 may be another method. For example, the road area 55 may be detected from the forward image 53 using a model that has been trained by machine learning.

(3) Other methods may be used to calculate the reliability. For example, the more noise there is in the front image 53, the lower the reliability R1 can be. Furthermore, the older the creation or update date of the map information 49 is, the lower the reliability R4 can be. Furthermore, the reliability may be determined depending on the method of estimating the forward gradient θ itself.

(4) The display control device 1 and its method according to the present disclosure are provided by configuring a processor and memory programmed to execute one or more functions embodied by a computer program. It may also be realized by a dedicated computer. Alternatively, the display control device 1 and its method described in the present disclosure may be implemented by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the display control device 1 and its method according to the present disclosure may include a processor configured with a processor and memory programmed to execute one or more functions, and one or more hardware logic circuits. It may also be realized by one or more dedicated computers configured in combination. The computer program may also be stored as instructions executed by a computer on a computer-readable non-transitory tangible storage medium. The method of realizing the functions of each part included in the display control device 1 does not necessarily need to include software, and all the functions may be realized using one or more pieces of hardware.

(5) Multiple functions of one component in the above embodiments may be realized by multiple components, and one function of one component may be realized by multiple components. . Further, a plurality of functions possessed by a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Further, a part of the configuration of the above embodiment may be omitted. Further, at least a part of the configuration of the above embodiment may be added to or replaced with the configuration of other embodiments.

(6) In addition to the display control device 1 described above, a system including the display control device 1 as a component, a program for making a computer function as the display control device 1, a semiconductor memory storing this program, etc. The present disclosure can also be realized in various forms such as a physical recording medium and a display control method.

DESCRIPTION OF SYMBOLS 1... Display control device, 3... Vehicle, 9... Image acquisition unit, 11... Upper end detection unit, 13... Gradient estimation unit, 15... Display position determination unit, 17... Display unit, 31... Upper end use unit, 33... Lane boundary Line usage unit, 47...HUD, 53...Front image, 55...Road area, 61...Top end, 71...Displayed object

Claims (3)

A display control device (1) that uses a display device (47) to display a display object (71) superimposed on a scenery (73) in front of a vehicle (3),
an image acquisition unit (9) configured to acquire an image (53) representing a scene in front of the vehicle;
a top edge detection unit (11) configured to detect a top edge (61) of a road area (55) in the image;
a slope estimation unit (13) configured to estimate the slope of the road;
a display position determining unit (15) configured to determine a vertical display position of the display object based on the slope estimated by the slope estimation unit;
a display unit (17) configured to use the display device to display the display object at the display position determined by the display position determining unit;
a lane boundary line detection unit (19) configured to extract edges, which are pixels whose brightness suddenly changes, in the image, and detect lane boundary lines based on the extracted edges;
a reliability calculation unit (27) configured to calculate reliability of the slope estimated by the slope estimation unit;
a clarity determining unit (29) configured to determine the clarity of a boundary between the displayed object and its surroundings according to the reliability;
Equipped with
The slope estimating unit is configured to estimate the slope of the road based on the position of the top edge in the image , and based on the direction in which the lane boundary line extends in the image, comprising a lane boundary usage unit (33) configured to estimate the slope of the road ;
The reliability calculation unit calculates the reliability higher as the number of edges extracted by the lane boundary detection unit increases.
Display control device. The display control device according to claim 1,
a position information acquisition unit (21) configured to acquire position information of the vehicle;
a reading unit (23) configured to read map information (49) including road slope information from a storage section (43);
Furthermore,
The slope estimating unit is a map using unit configured to estimate the slope of the road by comparing the position information acquired by the position information acquisition unit with the map information read by the reading unit. (35) further comprising:
Display control device. The display control device according to claim 1 or 2 ,
The vehicle further includes a target detection unit (25) configured to detect a target existing in front of the vehicle using a radar (45),
The slope estimation unit further includes a target object use unit (37) configured to estimate the slope of the road based on the direction of the target detected by the target detection unit.
Display control device.

Images