JP2021089578A - Display control device - Google Patents

Abstract

To provide a display control device that can apparently suppress a change in a distance between a display object and a road surface even when a road in front of a vehicle has a slope.SOLUTION: A display control device 1 uses a display unit to display a display object superimposed on a landscape in front of a vehicle. The display control device 1 includes: an image acquisition unit 9 that acquires an image; a top edge detection unit 11 that detects a top edge of a road area in the image; a slope estimation unit 13 that estimates a slope of a road; a display position determination unit 15 that determines, based on the slope, a display position of the display object in a vertical direction; and a display unit 17 that displays the display object at the display position determined by the display position determination unit. The slope estimation unit 13 estimates, based on the position of the top edge in the image, the slope of the road.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a display control device.

Patent Document 1 discloses a vehicle display device. The vehicle display device superimposes and displays a display object on an actual view such as a road surface.

WO2018 / 03320A1

There may be a slope on the road in front of the vehicle. If the display position of the display object is constant, the distance between the display object and the road surface of the road changes according to the slope. For example, suppose that the displayed object is displayed so as to touch the road surface when there is no gradient. If the display position of the display object is constant and there is a downward slope in front of the vehicle, the display object will be displayed floating from the road surface. Further, if the display position of the displayed object is constant and there is an upward slope in front of the vehicle, the displayed object is displayed as if it has sunk into the road surface.

In one aspect of the present disclosure, it is preferable to provide a display control device that can apparently suppress a change in the distance between the display object and the road surface even when the road in front of the vehicle has a slope.

One aspect of the present disclosure is a display control device (1) that uses a display device (47) to display a display object (71) superimposed on a landscape (73) in front of the vehicle (3). The display control device detects an image acquisition unit (9) configured to acquire an image (53) representing the landscape in front of the vehicle, and an upper end (61) of a road region (55) in the image. Based on the upper end detection unit (11) configured to perform the above, the gradient estimation unit (13) configured to estimate the slope of the road, and the gradient estimated by the gradient estimation unit, the display object. Using the display position determination unit (15) configured to determine the display position in the vertical direction and the display device, the display object is configured to be displayed at the display position determined by the display position determination unit. The display unit (17) is provided.

The gradient estimation unit includes an upper end use unit (31) configured to estimate the slope of the road based on the position of the upper end in the image.
The display control device, which is one aspect of the present disclosure, estimates the slope of the road. The display control device, which is one aspect of the present disclosure, determines the display position of the display object based on the estimated road slope. Therefore, according to the display control device, which is one aspect of the present disclosure, it is possible to apparently suppress a change in the distance between the display object and the road surface even when there is a slope of the road.

The display control device, which is one aspect of the present disclosure, estimates the slope of the road based on the position of the upper end of the area of the road. Therefore, the display control device, which is one aspect of the present disclosure, can easily estimate the slope of the road. Further, the display control device, which is one aspect of the present disclosure, can estimate the slope of the road even when another estimation method cannot be used.

It is a block diagram which shows the structure of a display control device. It is a block diagram which shows the functional structure of a display control device. It is a flowchart which shows the display process which a display control apparatus executes. It is a flowchart which shows the display process which a display control apparatus executes. It is a flowchart which shows the area detection process of the road executed by a display control device. It is explanatory drawing which shows the template and the front image. It is explanatory drawing which shows the front image, the area of a road, and the upper end. It is explanatory drawing which shows the method of estimating the forward gradient using the position of the upper end of the area of a road. It is explanatory drawing which shows the example of the display object.

An exemplary embodiment of the present disclosure will be described with reference to the drawings.
<First Embodiment>
1. 1. Configuration of Display Control Device 1 The configuration of the display control device 1 will be described with reference to FIGS. 1 and 2. As shown in FIG. 1, the display control device 1 is mounted on the vehicle 3. The display control device 1 includes a microcomputer having a CPU 5 and, for example, a semiconductor memory such as a RAM or a ROM (hereinafter referred to as a memory 7).

Each function of the display control device 1 is realized by the CPU 5 executing a program stored in a non-transitional substantive recording medium. In this example, the memory 7 corresponds to a non-transitional substantive recording medium in which a program is stored. Moreover, when this program is executed, the method corresponding to the program is executed. The display control device 1 may include one microcomputer or a plurality of microcomputers.

As shown in FIG. 2, the display control device 1 includes an image acquisition unit 9, an upper end detection unit 11, a gradient estimation unit 13, a display position determination unit 15, a display unit 17, and a lane boundary line detection unit 19. A position information acquisition unit 21, a reading unit 23, a target detection unit 25, a reliability calculation unit 27, and an intelligibility determination unit 29 are provided.

The gradient estimation unit 13 includes an upper end use unit 31, a lane boundary line use unit 33, a map use unit 35, and a target use unit 37.
The display control device 1 is connected to various configurations included in the vehicle 3. As shown in FIG. 1, as a configuration connected to the display control device 1, there are a camera 39, a GPS 41, a storage unit 43, a radar 45, and a head-up display (HUD) 47.

The camera 39 generates an image representing the landscape in front of the vehicle 3 (hereinafter referred to as a front image 53). The camera 39 sends the front image 53 to the display control device 1. GPS 41 acquires the position information of the vehicle 3.
The storage unit 43 stores the map information 49. The map information 49 includes gradient information at a plurality of places. Gradient information is information representing the slope of a road. The display control device 1 can read the map information 49.

The radar 45 can detect a target existing in front of the vehicle 3. Examples of the target include other vehicles, pedestrians, fixed objects, and the like. The radar 45 sends a signal indicating the detection result of the target to the display control device 1. The signal indicating the detection result of the target is a signal indicating the distance from the vehicle 3 to the target, the direction of the target with respect to the vehicle 3, and the like. The orientation of the target includes an orientation in the horizontal direction and an orientation in the vertical direction.

The HUD 47 displays a display object superimposed on the landscape in front of the vehicle 3. HUD47 corresponds to the display device. Examples of the display object include figures, letters, numbers, symbols, and the like. Examples of the figure include an arrow, a band-shaped figure, a straight line, a curve, a polygon, and the like.

The display object is, for example, a display object (hereinafter referred to as a virtual display object) that virtually appears to exist in a three-dimensional space when viewed from the driver of the vehicle 3. The virtual display object is, for example, a display object that virtually appears to have a constant distance from the road surface. The virtual display object is, for example, a display object that virtually extends along the road surface of the road.

2. Processes Executed by the Display Control Device 1 Display processes executed by the display control device 1 repeatedly at predetermined time intervals will be described with reference to FIGS. 3 to 9. In step 1 of FIG. 3, the image acquisition unit 9 acquires the front image 53 using the camera 39.

In step 2, the upper end detection unit 11 detects the road region 55 in the forward image 53 acquired in step 1. The process of detecting the area 55 of the road will be described with reference to FIGS. 5 and 6.
The upper end detection unit 11 includes N templates 51. N is a natural number of 1 or more. As shown in FIG. 6, the template 51 is smaller than the front image 53. Each of the N templates 51 is numbered from 1 to N. Each of the N templates 51 has a brightness similar to that of a general road. The brightness of the N templates 51 is different for each template.

In step 31, the upper end detection unit 11 sets the value of i to 1. i is the number of the template 51.
In step 32, the upper end detection unit 11 selects the i-th template 51.

In step 33, the upper end detection unit 11 superimposes the i-th template 51 on the initial position in the front image 53. The initial position is a preset position. The portion of the front image 53 that overlaps with the template 51 will be referred to as a comparison area below.

In step 34, the upper end detection unit 11 acquires the brightness of the comparison region.
In step 35, the upper end detection unit 11 determines whether or not the brightness of the template 51 and the brightness of the comparison region acquired in step 34 are close to each other. The close brightness means, for example, that the difference in brightness is equal to or less than a preset threshold value. When the brightness of the template 51 and the brightness of the comparison area are close to each other, this process proceeds to step 36. If the brightness of the template 51 and the brightness of the comparison area are not close to each other, this process proceeds to step 37.

In step 36, the upper end detection unit 11 assumes that the comparison area is a road area.
In step 37, the upper end detection unit 11 assumes that the comparison area is not a road area.
In step 38, the upper end detection unit 11 determines whether or not the position of the template 51 is the end position. The end position is the position of the template 51 when the template 51 has searched the entire area of the front image 53. If the position of the template 51 is not the end position, the process proceeds to step 39. If the position of the template 51 is the end position, this process proceeds to step 40.

In step 39, the top detection unit 11 moves the template 51 to the next position. After step 39, the process proceeds to step 34.
In step 40, the upper end detection unit 11 determines whether or not the value of i is N. When the value of i is N, it means that the processes of steps 32 to 39 have been executed for all the templates 51. When the value of i is N, the process of detecting the area of the road ends, and the process proceeds to step 3 of FIG. If the value of i is not N, this process proceeds to step 41.

In step 41, the upper end detection unit 11 increases the value of i by one. After step 41, the process proceeds to step 32.
As a result of the process of detecting the road area, as shown in FIG. 7, the road area 55 is detected in the front image 53. When performing the process of detecting the road area 55, the area where the vehicle or the structure is located may be excluded in advance from the front image 53, and the road area 55 may be detected in the area not excluded. .. The area where the vehicle or structure is located can be detected by the template matching method using the template of the vehicle or structure.

Returning to FIG. 3, in step 3, the lane boundary line detection unit 19 detects the lane boundary line in the front image 53 acquired in step 1. Examples of the lane boundary line include a white line and the like. The method for detecting the lane boundary line is, for example, as follows. The lane boundary line detection unit 19 extracts an edge in the front image 53. An edge is a pixel whose brightness changes suddenly. The lane boundary line detection unit 19 detects the lane boundary line by a known method based on the extracted edge.

In step 4, the lane boundary line use unit 33 estimates the slope of the road ahead of the vehicle 3 (hereinafter referred to as the forward slope θ) based on the lane boundary line detected in step 3. As shown in FIG. 8, the forward gradient θ is the gradient of the road 59 in front of the vehicle 3 with reference to the road 57 on which the vehicle 3 is located.

In the front image 53, the direction in which the lane boundary line extends and the forward gradient θ have a correlation. The lane boundary line use unit 33 estimates the forward gradient θ based on the direction in which the lane boundary line extends in the front image 53.

In step 5, the reliability calculation unit 27 calculates the reliability R1 of the forward gradient θ estimated in step 4. The reliability calculation unit 27 calculates the reliability R1 higher as the number of edges extracted in step 3 increases.

In step 6, the reliability calculation unit 27 determines whether or not the reliability R1 calculated in step 5 is equal to or higher than a preset threshold value. When the reliability R1 is equal to or higher than the threshold value, this process proceeds to step 22 in FIG. If the reliability R1 is less than the threshold value, this process proceeds to step 7.

In step 7, the upper end detection unit 11 detects the upper end 61 shown in FIG. 7 in the front image 53 acquired in step 1. The upper end 61 is the upper end of the road area 55.
When the upper end 61 is displayed in the front image 53, the upper end detection unit 11 directly detects the upper end 61. In the front image 53, when the upper end 61 is hidden by a target or the like, the upper end detection unit 11 extends the left and right boundary lines 63 and 65 of the road region 55 forward. The upper end detection unit 11 has an upper end 61 at the intersection of the boundary line 63 and the boundary line 65.

In step 8, the upper end use unit 31 estimates the forward gradient θ based on the position of the upper end 61 detected in step 7. The method of estimating the forward gradient θ will be described below. As shown in FIG. 8, the vanishing point when the forward gradient θ is 0 degrees is defined as the reference vanishing point 67. The reference vanishing point 67 is at a fixed position within the angle of view of the camera 39. Θ'shown in FIG. 8 is represented by the following equation (1).

In the formula (1), d is the distance between the upper end 61 and the reference vanishing point 67 at the imaging position 69 of the camera 39. F in the equation (1) is the focal length of the camera 39. The upper end use unit 31 calculates θ'by the equation (1). θ'is approximated by the forward gradient θ. Therefore, the upper end use unit 31 sets the calculated θ'as the forward gradient θ.

In step 9, the reliability calculation unit 27 calculates the reliability R2 of the forward gradient θ estimated in step 8. The reliability calculation unit 27 calculates a statistical value of the correlation value of the brightness over the entire area 55 of the road. The luminance correlation value is a value representing the closeness between the luminance of the template 51 and the luminance of the comparison region compared in step 35. The closer the brightness of the template 51 to the brightness of the comparison area, the larger the correlation value. The reliability calculation unit 27 calculates the reliability R2 higher as the statistical value becomes larger.

In step 10, the reliability calculation unit 27 determines whether or not the reliability R2 calculated in step 9 is equal to or higher than a preset threshold value. When the reliability R2 is equal to or higher than the threshold value, this process proceeds to step 22 in FIG. If the reliability R2 is less than the threshold value, this process proceeds to step 11.

In step 11, the target detection unit 25 uses the radar 45 to execute a process of detecting a target in front of the vehicle 3.
In step 12, the target detection unit 25 determines whether or not the target has been detected in the process of step 11. If a target is detected, this process proceeds to step 13. If no target is detected, this process proceeds to step 16.

In step 13, the target use unit 37 acquires the direction of the target detected in the process of step 11. The direction of the target is the direction with respect to the vehicle 3. There is a correlation between the direction of the target and the forward gradient θ. For example, the forward gradient θ is an ascending gradient, and the larger the value of the forward gradient θ, the higher the direction of the target. The target use unit 37 estimates the forward gradient θ based on the direction of the target.

When a plurality of targets are detected in the process of step 11, the target use unit 37 first estimates the forward gradient θ for each target. Next, the target use unit 37 estimates the final forward slope θ by statistically processing the forward slope θ for each target.

In step 14, the reliability calculation unit 27 calculates the reliability R3 of the forward gradient θ estimated in step 13. The reliability calculation unit 27 calculates the reliability R3 higher as the number of targets detected in the process of step 11 increases.

In step 15, the reliability calculation unit 27 determines whether or not the reliability R3 calculated in step 14 is equal to or higher than a preset threshold value. When the reliability R3 is equal to or higher than the threshold value, this process proceeds to step 22 in FIG. If the reliability R3 is less than the threshold value, this process proceeds to step 16.

In step 16, the position information acquisition unit 21 acquires the position information of the vehicle 3 by using the GPS 41.
In step 17, the reading unit 23 determines whether or not the map information 49 can be read from the storage unit 43. If the map information 49 can be read from the storage unit 43, this process proceeds to step 18 of FIG. If the map information 49 cannot be read from the storage unit 43, this process proceeds to step 21 of FIG.

In step 18, first, the reading unit 23 reads the map information 49 from the storage unit 43. Next, the map use unit 35 estimates the forward gradient θ by collating the position information of the vehicle 3 acquired in step 16 with the read map information 49.

In step 19, the reliability calculation unit 27 calculates the reliability R4 of the forward gradient θ estimated in step 18. The reliability calculation unit 27 calculates the reliability R4 higher as the reliability value of the position information of the vehicle 3 acquired in step 16 is higher. The reliability value of the position information of the vehicle 3 becomes higher, for example, as the number of GPS satellites that can receive the signal increases.

In step 20, the reliability calculation unit 27 determines whether or not the reliability R4 calculated in step 19 is equal to or higher than a preset threshold value. If the reliability R4 is equal to or higher than the threshold value, the present process proceeds to step 22. If the reliability R4 is less than the threshold value, this process proceeds to step 21.

In step 21, the gradient estimation unit 13 sets the value of the forward gradient θ as the value estimated in the previous display process.
In step 22, the display position determination unit 15 determines the display position in the vertical direction of the display object based on the forward gradient θ. The forward gradient θ to be used is the forward gradient θ estimated in step 4 when the affirmative judgment is made in step 6.

Further, the forward gradient θ to be used is the forward gradient θ estimated in step 8 when affirmatively determined in step 10. Further, the forward gradient θ to be used is the forward gradient θ estimated in step 13 when affirmatively determined in step 15.

Further, the forward gradient θ to be used is the forward gradient θ estimated in step 18 when affirmatively determined in step 20. Further, the forward gradient θ to be used is the forward gradient θ estimated in the previous display process when a negative determination is made in step 20.

The method by which the display position determination unit 15 determines the display position is as follows. The display position determining unit 15 defines a plane having a pitch angle corresponding to the forward gradient θ in the three-dimensional space. The display position determining unit 15 virtually arranges a display object on the plane in the three-dimensional space. The display position determining unit 15 converts a plane and a display object in the three-dimensional space into an image in the two-dimensional space. The display position determining unit 15 adopts the position of the displayed object in the converted image.

In step 23, the intelligibility determination unit 29 determines the intelligibility of the boundary between the display object and its surroundings according to the reliability. The reliability used is the reliability of the forward gradient θ used in the process of step 22. For example, the higher the reliability of the clarity determination unit 29, the higher the clarity of the boundary. Examples of the mode in which the boundary clarity is low include a mode in which the boundary is gradation, a mode in which a part or all of the displayed object is highly transparent, and the like.

In step 24, the display unit 17 displays a display object using the HUD 47. The display position is the position determined in step 22.
FIG. 9 shows an example of the display object 71. The display object 71 is superimposed on the landscape 73 in front of the vehicle 3 when viewed from the viewpoint of the driver of the vehicle 3. From the driver's point of view, the display object 71 virtually appears as if it exists in a three-dimensional space. The display object 71 extends along the road surface of the road 59 ahead. The display object 71 is a display object in which the distance from the road surface of the road 59 in front appears to be constant regardless of the presence or absence of the forward gradient θ and the magnitude of the forward gradient θ.

3. 3. Effect of display control device 1 (1A) The display control device 1 estimates the forward gradient θ. The display control device 1 determines the display position of the display object based on the estimated forward gradient θ. Therefore, according to the display control device 1, the display object can be displayed so that the distance between the display object and the road surface of the road can be seen to be constant even when there is a forward gradient θ.

The display control device 1 estimates the forward gradient θ based on the position of the upper end 61 in the forward image 53. Therefore, the display control device 1 can easily estimate the forward gradient θ. Further, the display control device 1 can estimate the forward gradient θ even when another estimation method cannot be used.

(1B) The display control device 1 detects the lane boundary line in the front image 53. The display control device 1 estimates the forward gradient θ based on the direction in which the lane boundary line extends in the front image 53. Therefore, the display control device 1 can estimate the forward gradient θ more accurately. Further, the display control device 1 can estimate the forward gradient θ even when another estimation method cannot be used.

(1C) The display control device 1 acquires the position information of the vehicle 3. The display control device 1 reads out the map information 49 from the storage unit 43. The display control device 1 estimates the forward gradient θ by collating the position information of the vehicle 3 with the map information 49. Therefore, the display control device 1 can estimate the forward gradient θ more accurately. Further, the display control device 1 can estimate the forward gradient θ even when another estimation method cannot be used.

(1D) The display control device 1 detects a target existing in front of the vehicle 3 by using the radar 45. The display control device 1 estimates the forward gradient θ based on the detected direction of the target. Therefore, the display control device 1 can estimate the forward gradient θ more accurately. Further, the display control device 1 can estimate the forward gradient θ even when another estimation method cannot be used.

(1E) The display control device 1 calculates the reliability of the estimated forward gradient θ. The display control device 1 determines the clarity of the boundary between the display object and its surroundings according to the reliability. Therefore, when the reliability is low, the clarity of the boundary of the displayed object is low. As a result, even if there is a deviation between the displayed object and the road surface, the deviation becomes less noticeable.
<Other Embodiments>
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and can be implemented in various modifications.

(1) The display device for displaying the display object may be a display device other than HUD47. For example, it may be a display device in which a display element is embedded in a windshield.
(2) The detection method for detecting the area 55 of the road may be another method. For example, the road region 55 may be detected from the front image 53 using a model trained by machine learning.

(3) The method for calculating the reliability may be another method. For example, the more noise the front image 53 has, the lower the reliability R1 can be. Further, the older the date of creation or update of the map information 49, the lower the reliability R4 can be. Further, the reliability may be determined according to the method of estimating the forward gradient θ itself.

(4) The display control device 1 and its method described in the present disclosure are provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. It may be realized by a dedicated computer. Alternatively, the display control device 1 and its method described in the present disclosure may be realized by a dedicated computer provided by configuring the processor with one or more dedicated hardware logic circuits. Alternatively, the display control device 1 and its method described in the present disclosure include a processor and memory programmed to perform one or more functions and a processor composed of one or more hardware logic circuits. It may be realized by one or more dedicated computers configured by a combination. The computer program may also be stored on a computer-readable non-transitional tangible recording medium as an instruction executed by the computer. The method for realizing the functions of each part included in the display control device 1 does not necessarily include software, and all the functions may be realized by using one or a plurality of hardware.

(5) A plurality of functions possessed by one component in the above embodiment may be realized by a plurality of components, or one function possessed by one component may be realized by a plurality of components. .. Further, a plurality of functions possessed by the plurality of components may be realized by one component, or one function realized by the plurality of components may be realized by one component. Further, a part of the configuration of the above embodiment may be omitted. In addition, at least a part of the configuration of the above embodiment may be added or replaced with the configuration of the other above embodiment.

(6) In addition to the above-mentioned display control device 1, a system having the display control device 1 as a component, a program for operating a computer as the display control device 1, and a non-transitional semiconductor memory or the like in which this program is recorded The present disclosure can also be realized in various forms such as an actual recording medium and a display control method.

1 ... Display control device, 3 ... Vehicle, 9 ... Image acquisition unit, 11 ... Upper end detection unit, 13 ... Gradient estimation unit, 15 ... Display position determination unit, 17 ... Display unit, 31 ... Upper end use unit, 33 ... Lane boundary Line use unit, 47 ... HUD, 53 ... forward image, 55 ... road area, 61 ... top edge, 71 ... display

Claims (5)

A display control device (1) for displaying a display object (71) superimposed on a landscape (73) in front of a vehicle (3) using a display device (47).
An image acquisition unit (9) configured to acquire an image (53) representing the landscape in front of the vehicle, and an image acquisition unit (9).
In the image, the upper end detection unit (11) configured to detect the upper end (61) of the road area (55), and the upper end detection unit (11).
A gradient estimation unit (13) configured to estimate the slope of the road, and
A display position determination unit (15) configured to determine a display position in the vertical direction of the display object based on the gradient estimated by the gradient estimation unit.
A display unit (17) configured to display the display object at the display position determined by the display position determining unit using the display device.
With
The gradient estimation unit includes an upper end use unit (31) configured to estimate the slope of the road based on the position of the upper end in the image.
Display control device. The display control device according to claim 1.
In the image, the lane boundary detection unit (19) configured to detect the lane boundary is further provided.
The gradient estimation unit
A lane boundary use unit (33) configured to estimate the slope of the road based on the direction in which the lane boundary extends in the image is further provided.
Display control device. The display control device according to claim 1 or 2.
A position information acquisition unit (21) configured to acquire the position information of the vehicle, and
A reading unit (23) configured to read the map information from a storage unit (43) that stores map information (49) including road gradient information.
With more
The gradient estimation unit is a map use unit configured to estimate the slope of the road by collating the position information acquired by the position information acquisition unit with the map information read by the reading unit. Further equipped with (35),
Display control device. The display control device according to any one of claims 1 to 3.
A target detection unit (25) configured to detect a target existing in front of the vehicle using a radar (45) is further provided.
The gradient estimation unit further includes a target use unit (37) configured to estimate the slope of the road based on the orientation of the target detected by the target detection unit.
Display control device. The display control device according to any one of claims 1 to 4.
A reliability calculation unit (27) configured to calculate the reliability of the gradient estimated by the gradient estimation unit, and
An intelligibility determination unit (29) configured to determine the intelligibility of the boundary between the display object and its surroundings according to the reliability.
A display control device further equipped with.

Images