JP6720683B2 - Vehicle display device and vehicle display method - Google Patents

Description

The present invention relates to a vehicle display technology relates to a vehicle display device and the vehicle display method in particular display an image.

The vehicle is equipped with multiple out-of-vehicle cameras, the images taken by these cameras are combined, and a bird's-eye view image of the surroundings of the vehicle is formed on the monitor as if the camera was installed above the vehicle. Is displayed. In such a bird's-eye view image display, it is necessary to intuitively recognize the distinction between a three-dimensional object and a two-dimensional object that is an obstacle because the three-dimensional object that is an obstacle cannot be displayed accurately due to the viewpoint conversion processing. Is difficult. Therefore, a three-dimensional object having a predetermined height or more is displayed in a mode different from the others (see, for example, Patent Document 1).

JP, 2009-49943, A

The images captured by the image capturing units installed on the four sides of the vehicle are subjected to the viewpoint conversion processing so as to look down from above the vehicle, thereby generating an overhead image. Even if there is no obstacle around the vehicle, if there is a step on the road surface around the vehicle, it is difficult to grasp the presence or absence of the step from the overhead view image.

The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for grasping the presence/absence of a step even in an overhead image.

In order to solve the above problems, a vehicular display device according to an aspect of the present invention includes a video acquisition unit that acquires a video captured around the vehicle, and a video acquired from the video acquisition unit from above the vehicle. By converting the viewpoint as seen, a generation unit that generates an overhead view image, a driving condition acquisition unit that acquires the driving condition of the vehicle, and a driving condition that the driving condition acquired by the driving condition acquisition unit is less than or equal to a predetermined speed. When the vehicle is stopped, the road surface shape acquisition unit that acquires the road surface shape around the vehicle and the road surface that determines whether or not there is a road surface shape that requires attention based on the road surface shape acquired by the road surface shape acquisition unit When the shape determination unit and the road surface shape determination unit determine that there is a road surface shape that requires attention, the image processing unit that superimposes the image showing the road surface shape that requires attention on the overhead view image, and the image processing unit that superimposes the image. And a display control unit for displaying a bird's-eye view image on the display unit.
Another aspect of the present invention is also a display device for a vehicle. This device generates a bird's-eye view image by converting the viewpoint of the image acquired by the image acquisition unit that acquires an image of the surroundings of the vehicle and the image acquired by the image acquisition unit as seen from above the vehicle. Based on the generation unit, the road surface shape acquisition unit that acquires the road surface shape around the vehicle, and the road surface shape acquired by the road surface shape acquisition unit, as a road surface shape that requires attention, If the road surface shape determination unit that determines whether or not a certain road surface shape exists and the road surface shape determination unit determines that there is a road surface shape that requires attention, an image showing the road surface shape that requires attention is superimposed on the overhead view video. An image processing unit and a display control unit that causes the display unit to display a bird's-eye view image on which images are superimposed in the image processing unit.
Yet another aspect of the present invention is also a vehicular display device. This device generates a bird's-eye view image by converting the viewpoint of the image acquired by the image acquisition unit that acquires an image of the surroundings of the vehicle and the image acquired by the image acquisition unit as seen from above the vehicle. As a road surface shape that requires attention based on the generation unit, the road surface shape acquisition unit that acquires the road surface shape around the vehicle, and the road surface shape acquired by the road surface shape acquisition unit, it becomes difficult to open the vehicle door. A road surface shape determination unit that determines whether or not there is a road surface shape with a height difference, and if the road surface shape determination unit determines that there is a road surface shape that requires attention, a bird's-eye view image of an image showing the road surface shape that requires attention And a display control unit that causes the display unit to display the bird's-eye view image on which the image is superimposed in the video processing unit.

Yet another aspect of the present invention is a vehicle display method. This method includes the steps of obtaining an image of the surroundings of the vehicle, generating a bird's-eye view image by converting the viewpoint of the obtained image as seen from above the vehicle, and running the vehicle. The step of acquiring the situation, the step of acquiring the road surface shape in the vicinity of the vehicle when the acquired traveling status is traveling or stopping at a predetermined speed , and the road surface requiring attention based on the acquired road surface shape. A step of determining whether a shape exists, a step of superimposing an image showing a road surface shape requiring attention on the overhead view video when it is determined that a road surface shape requiring attention exists, and an overhead view video on which the image is superimposed Is displayed on the display unit.
Yet another aspect of the present invention is also a display method for a vehicle. This method includes the steps of obtaining an image of the surroundings of the vehicle, generating a bird's-eye view image by converting the viewpoint of the obtained image as seen from above the vehicle, and the periphery of the vehicle. And a step of determining whether or not there is a road surface shape having a height difference that makes it difficult for the vehicle to travel, as a road surface shape that requires attention based on the acquired road surface shape. When it is determined that a road surface shape requiring attention exists, the method includes: superimposing an image showing the road surface shape requiring attention on the overhead view video; and displaying the overhead view video on which the image is superimposed on the display unit.
Yet another aspect of the present invention is also a display method for a vehicle. This method includes the steps of obtaining an image of the surroundings of the vehicle, generating a bird's-eye view image by converting the viewpoint of the obtained image as seen from above the vehicle, and the periphery of the vehicle. In the step of acquiring the road surface shape in step 1, and based on the acquired road surface shape, it is determined whether there is a road surface shape with a height difference that makes it difficult to open the vehicle door as a road surface shape that requires attention. Steps, when it is determined that there is a road surface shape requiring attention, a step of superimposing an image showing the road surface shape requiring attention on the bird's-eye view image, and a step of displaying the bird's-eye view image on which the image is superimposed on the display unit. Prepare

It should be noted that any combination of the above constituent elements, and the expression of the present invention converted between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

According to the present invention, the presence/absence of a step can be grasped even in a bird's eye view image.

FIG. 3 is a perspective view showing an imaging range formed around the vehicle according to the first embodiment of the present invention. It is a figure which shows the sensor mounted in the vehicle of FIG. It is a figure which shows the structure of the display apparatus for vehicles mounted in the vehicle of FIG. It is a figure which shows the overhead view image produced|generated in the production|generation part of FIG. It is a figure which shows arrangement|positioning of the front sensor of FIG. It is a figure which shows arrangement|positioning of the left side sensor of FIG. It is a figure which shows the outline of a process in the road surface shape acquisition part of FIG. It is a figure which shows the outline of another process in the road surface shape acquisition part of FIG. It is a figure which shows the outline|summary of another process in the road surface shape acquisition part of FIG. It is a figure which shows the data structure of the table memorize|stored in the road surface shape determination part of FIG. It is a figure which shows the bird's-eye view image produced|generated in the image processing part of FIG. It is a figure which shows another overhead view image produced|generated in the image processing part of FIG. It is a figure which shows another overhead view image produced|generated in the image processing part of FIG. It is a figure which shows another overhead view image produced|generated in the image processing part of FIG. It is a figure which shows another overhead view image produced|generated in the image processing part of FIG. FIGS. 16A and 16B are diagrams showing images generated by the image processing unit in FIG. 4 is a flowchart showing a display procedure by the vehicle display device of FIG. 3. It is a figure which shows the structure of the display device for vehicles which concerns on Example 2 of this invention. It is a figure which shows the outline|summary of the process in the road surface shape acquisition part of FIG. It is a figure which shows the data structure of the table memorize|stored in the road surface shape determination part of FIG. It is a figure which shows the bird's-eye view image produced|generated in the image processing part of FIG. It is a figure which shows another overhead view image produced|generated in the image processing part of FIG.

(Example 1)
Before specifically describing the present invention, the premise will be described first. Example 1 of the present invention relates to a vehicular display device that generates a bird's-eye view image by viewpoint conversion with respect to images picked up by a plurality of image pickup units installed in a vehicle, and displays the generated bird's-eye view image. In the viewpoint conversion, since the captured image is mapped on the plane on the ground, the objects included in the captured image are displayed as if they are aligned on the plane of the ground. Therefore, even if there is a step on the road surface around the vehicle, such a step is planarly shown in the overhead view image. As a result, it is difficult for the driver to grasp the presence/absence of a step while looking at the overhead image. In particular, the step that becomes lower in the direction away from the vehicle side is more difficult to grasp because the vertical surface of the step is not imaged by the imaging unit.

To cope with this, the vehicular display device according to the present embodiment executes the following processing. A plurality of sensors are installed in the vehicle, and the vehicle display device acquires the road surface shape around the vehicle based on the detection results of the plurality of sensors. If the acquired road surface shape includes a road surface shape that requires a surrounding area, the vehicle display device displays a bird's-eye view image on which an image indicating the road surface shape is superimposed. Here, the road surface shape that requires the periphery corresponds to, for example, a step existing on the road surface.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Specific numerical values and the like shown in the examples are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the specification and the drawings, elements having substantially the same function and configuration are denoted by the same reference numerals to omit redundant description, and elements not directly related to the present invention are omitted. ..

FIG. 1 is a perspective view showing an imaging range formed around a vehicle 100 according to a first embodiment of the present invention. A front imaging unit 12 is installed in a front portion of the vehicle 100, such as a bumper and a hood. The front imaging unit 12 forms a front imaging region 20 so as to face forward from the front imaging unit 12, and captures an image in the front imaging region 20. A left side image pickup unit 14 is installed in a left side portion of the vehicle 100, for example, a lower portion of a left side door mirror. The left side image capturing unit 14 forms a left side image capturing region 22 from the left side image capturing unit 14 toward the left, and captures an image in the left side image capturing region 22.

A rear imaging unit 16 is installed in a rear portion of the vehicle 100, such as a bumper or a trunk. The rear imaging unit 16 forms a rear imaging region 24 from the rear imaging unit 16 toward the rear, and captures an image in the rear imaging region 24. A right side image pickup unit 18 (not shown) is installed in the right side portion of the vehicle 100 so as to be symmetrical with the left side image pickup unit 14. The right side image capturing section 18 forms a right side image capturing area 26 from the right side image capturing section 18 toward the right side, and captures an image in the right side image capturing area 26. The front image capturing unit 12, the left side image capturing unit 14, the rear image capturing unit 16, and the right side image capturing unit 18 are used as the image capturing unit 10, and the image captured by the image capturing unit 10 captures the periphery of the vehicle 100.

FIG. 2 shows a sensor mounted on the vehicle 100. A front sensor 112, a left side sensor 114, a rear sensor 116, a right side sensor 118, a left front sensor 120, a right front sensor 122, a left rear sensor 124, and a right rear sensor 126 are arranged in the vehicle 100. The left front sensor 120, the front sensor 112, and the right front sensor 122 are installed in the front portion of the vehicle 100. Here, the left front sensor 120 is arranged on the left side, the front sensor 112 is arranged on the center, and the right front sensor 122 is arranged on the right side. The left side sensor 114 is arranged in the left side portion of the vehicle 100, and the right side sensor 118 is arranged in the right side portion of the vehicle 100. The left rear sensor 124, the rear sensor 116, and the right rear sensor 126 are installed in the rear portion of the vehicle 100. Here, the left rear sensor 124 is arranged on the left side, the rear sensor 116 is arranged at the center, and the right rear sensor 126 is arranged on the right side.

FIG. 3 shows a configuration of the vehicle display device 30 mounted on the vehicle 100. The vehicular display device 30 is connected to the imaging unit 10, the display unit 50, and the sensor 110. The image capturing section 10 includes a front image capturing section 12, a left side image capturing section 14, a rear image capturing section 16, and a right side image capturing section 18, and a sensor 110 includes a front sensor 112, a left side sensor 114, a rear sensor 116, and a right side sensor 118. , A front left sensor 120, a front right sensor 122, a rear left sensor 124, and a rear right sensor 126. The vehicle display device 30 includes an image acquisition unit 32, a generation unit 34, an image processing unit 40, a display control unit 42, a traveling condition acquisition unit 44, a road surface shape acquisition unit 46, and a road surface shape determination unit 48.

The front image pickup unit 12, the left side image pickup unit 14, the rear image pickup unit 16, and the right side image pickup unit 18 pick up images as described above. The front image capturing unit 12, the left side image capturing unit 14, the rear image capturing unit 16, and the right side image capturing unit 18 output the image to the image obtaining unit 32. The video acquisition unit 32 acquires video from each of the front imaging unit 12, the left side imaging unit 14, the rear imaging unit 16, and the right side imaging unit 18. That is, the image acquisition unit 32 acquires an image of the surroundings of the vehicle 100. The video acquisition unit 32 outputs the acquired video to the generation unit 34.

The generation unit 34 inputs the video from the video acquisition unit 32. The generation unit 34 generates a bird's-eye view image by performing a process of converting a viewpoint on the image as viewed from above the vehicle 100. A known technique may be used for the conversion. For example, each pixel of the image is projected on a three-dimensional curved surface in a virtual three-dimensional space, and the three-dimensional image is displayed according to a virtual viewpoint from above the vehicle 100. Cut out the required area of the curved surface. The cut-out area corresponds to an image obtained by converting the viewpoint.

FIG. 4 shows a bird's-eye view image 78 generated by the generation unit 34. A vehicle icon 80, which is an image showing the vehicle 100, is arranged in the central portion of the overhead view video 78. The own vehicle icon 80 is an image of the vehicle 100 viewed from above and has the size of the vehicle 100 in the overhead view video 78. A front image 70 is arranged in front of the own vehicle icon 80, a left side image 72 is arranged on the left side of the own vehicle icon 80, a rear image 74 is arranged behind the vehicle 100, and a right side of the vehicle 100. A right side image 76 is arranged on the right side. Return to FIG. The generation unit 34 outputs the overhead view video 78 to the video processing unit 40.

The traveling status acquisition unit 44 acquires information on the traveling status from an ECU (Electronic Control Unit) via a CAN (Controller Area Network). The information on the traveling situation is, for example, information on the traveling speed of the vehicle 100. The traveling condition acquisition unit 44 outputs information regarding the traveling condition to the road surface shape acquisition unit 46.

The front sensor 112, the left side sensor 114, the rear side sensor 116, the right side sensor 118, the left front side sensor 120, the right front side sensor 122, the left rear side sensor 124, and the right rear side sensor 126 are integrated as a sensor 110. As the sensor 110, existing technologies such as a sensor using infrared laser light and a sensor using ultrasonic waves can be applied. When the infrared laser light is used for the sensor 110, the infrared laser light is scanned in a predetermined range by a scanning mirror or the like, and the distance to the road surface is acquired from the phase difference of the reflected laser light. When an ultrasonic wave is used for the sensor 110, an ultrasonic pulse is output in a predetermined range, and the distance to the road surface is acquired from the time difference from the reflected wave.

FIG. 5 shows the arrangement of the front sensor 112. The front sensor 112 is used to detect whether a hole or a cliff exists in the traveling direction. The front sensor 112 is arranged at a height of “A1” from the road surface 90, and is arranged so as to mainly detect a point away from the vehicle 100 by “B1”. For example, the front sensor 112 is arranged at a position near the hood at the front end of the vehicle 100. Also, "B1" is set to 2 m. As shown in the drawing, when the groove 92 having the width “t” is present at a point “B1” away from the vehicle 100, the wall surface of the groove 92 is detected. The left front sensor 120, the right front sensor 122, the left rear sensor 124, the rear sensor 116, and the right rear sensor 126 of FIG. 2 are similarly arranged. The front left sensor 120, the front right sensor 122, the rear left sensor 124, and the rear right sensor 126 are used to detect whether or not there is a hole or a cliff at the bended point of the vehicle 100.

FIG. 6 shows the arrangement of the left side sensor 114. The left side sensor 114 is used to detect whether a hole or a cliff exists on the left side of the vehicle 100. The left side sensor 114 is also used to detect whether or not a hole or a cliff exists at a position where an occupant of the vehicle 100 gets off from the left side of the vehicle 100. The left side sensor 114 is arranged at a height of “A2” from the road surface 90, and is arranged so as to mainly detect a point away from the vehicle 100 by “B2”. For example, the left side sensor 114 is located on the door or on the door mirror. Also, "B2" is set to 10 cm. As shown in the figure, when the groove 92 having the width “t” exists at a point “B2” away from the vehicle 100, the bottom surface of the groove 92 is detected. Thus, the higher the position of the left side sensor 114 is, the easier the depth of the groove 92 is detected. Return to FIG. The sensor 110 outputs the detection result to the road surface shape acquisition unit 46.

The road surface shape acquisition unit 46 inputs the detection result from the sensor 110. The road surface shape acquisition unit 46 acquires data indicating the road surface shape around the vehicle 100 by analyzing the detection result from the sensor 110. The data indicating the road surface shape includes position information of the detection range and distance information from the sensor 110. More specifically, the road surface shape acquisition unit 46 acquires the road surface shape by acquiring the distance from the vehicle body of the vehicle 100 to the road surface within the range in which the road surface shape can be acquired. FIG. 7 shows an outline of processing in the road surface shape acquisition unit 46. Here, for the sake of clarity, the left side sensor 114 is shown as the sensor 110, and only the acquisition of the road surface shape in the left and right direction of the vehicle 100 by the left side sensor 114 is illustrated. Note that, in reality, the road surface shape in the front-rear direction of the vehicle 100 is also acquired by the left side sensor 114, and the road surface shape acquired in the left-right direction and the road surface shape acquired in the front-rear direction. Are combined.

The left side image pickup unit 14 and the left side sensor 114 are attached to the left side door mirror of the vehicle 100. The left side image capturing unit 14 captures an image in the left side image capturing area 22, and the left side sensor 114 measures a distance in the left side detecting area 130. The left side detection area 130 is a range in which the road surface shape can be acquired by the left side sensor 114, and is shown as an area “PA” from the point P1 to the point P2. In addition, points P3 and P4 are shown as intermediate points included in the area “PA”. The intermediate points are not limited to the points P3 and P4. Furthermore, the distance L1 with respect to the point P1, the distance L2 with respect to the point P2, the distance L3 with respect to the point P3, and the distance L4 with respect to the point P4 are shown, respectively. Under such circumstances, the data indicating the road surface shape includes the combination of the point P1 and the distance L1, the combination of the point P2 and the distance L2, the combination of the point P3 and the distance L3, and the combination of the point P4 and the distance L4. Is included.

FIG. 8 shows an outline of another process in the road surface shape acquisition unit 46. Here, there is a low step portion 94 that is "h" lower than the road surface 90. As shown, a distance L1' to the point P1, a distance L2' to the point P2, a distance L3' to the point P3, and a distance L4' to the point P4 are shown. Therefore, in the data indicating the road surface shape, there are a combination of the point P1 and the distance L1′, a combination of the point P2 and the distance L2′, a combination of the point P3 and the distance L3′, and a combination of the point P4 and the distance L4′. included. With such data, the road surface shape as shown in FIG. 8 is shown.

FIG. 9 shows an outline of still another process in the road surface shape acquisition unit 46. Here, there is a low step portion 94 that is "h" lower than the road surface 90. Note that “h” in FIG. 9 is assumed to be larger than “h” in FIG. That is, the step in FIG. 9 is larger than the step in FIG. As illustrated, a distance L1″ for the point P1, a distance L2″ for the point P2, a distance L3″ for the point P3, and a distance L4″ for the point P4 are shown. Therefore, in the data showing the road surface shape, the combination of the point P1 and the distance L1″, the combination of the point P2 and the distance L2″, the combination of the point P3 and the distance L3″, the point P4 and the distance L4″. In combination with. Such data shows the road surface shape as shown in FIG. Return to FIG.

The road surface shape acquisition unit 46 inputs information on the traveling condition from the traveling condition acquisition unit 44. The road surface shape acquisition unit 46 acquires data indicating the road surface shape, as described above, when the traveling condition is traveling or stopping at a speed lower than the predetermined speed. This is equivalent to acquiring the road surface shape around the vehicle 100. On the other hand, the road surface shape acquisition unit 46 does not acquire the data indicating the road surface shape when the traveling condition is traveling at a speed higher than the predetermined speed. When the road surface shape acquisition unit 46 acquires the data indicating the road surface shape, the road surface shape acquisition unit 46 outputs the data indicating the road surface shape to the road surface shape determination unit 48.

The road surface shape determination unit 48 inputs data indicating the road surface shape from the road surface shape acquisition unit 46. The road surface shape determination unit 48 determines whether or not there is a road surface shape requiring attention based on the road surface shape indicated in the data input from the road surface shape acquisition unit 46. The road surface shape that needs attention is, for example, a groove 92 having a width equal to or larger than a predetermined value, or a step having a height difference equal to or larger than a predetermined value. Note that the groove 92 is the same as the descending step when the width is increased. Here, in order to determine whether or not there is a road surface shape requiring attention, the road surface shape determination unit 48 stores a reference pattern of the road surface shape requiring attention in the road surface shape as a table.

FIG. 10 shows a data structure of a table stored in the road surface shape determination unit 48. When the difference in distance between adjacent points is equal to or larger than the threshold value, it is recognized that there is a step on the road surface 90. Further, the difference at that time is acquired as the height h. The height h is a parameter when the road surface shape is a down step or an up step. If the road surface shape is the groove 92, the groove width t is used as a parameter. When the difference in the distance L between the adjacent points is smaller than the threshold value, the road surface 90 is recognized as a flat surface or a slope. In that case, the process in the road surface shape determination unit 48 is not executed. In this way, it is recognized that there is a step when the distance L changes discretely, and there is no step when the distance L changes continuously.

Depending on the value of the parameter, "safety", "caution driving", and "running prohibited" are defined. The range of parameter values for “safety”, “caution driving”, and “running prohibited” is determined based on the shape of the vehicle 100, particularly the minimum ground clearance and tire width. In this way, the road surface shape determination unit 48 determines whether or not there is a road surface shape with a height difference that makes it difficult for the vehicle 100 to travel as a road surface shape that requires attention. In particular, the road surface shape determination unit 48 determines whether or not there is a step shape that becomes lower when the vehicle 100 is separated from the vehicle 100 as a road surface shape having a height difference that makes traveling of the vehicle 100 difficult. Return to FIG.

The road surface shape determination unit 48 refers to the table shown in FIG. 10, and based on the road surface shape shown in the data input from the road surface shape acquisition unit 46, “safety”, “caution traveling”, “running impossible”. Identify one of the. At that time, the road surface shape determination unit 48 also specifies the position information of the step or the groove 92 that specifies any of "safety", "careful driving", and "running impossible". The road surface shape determination unit 48 makes an image of information (hereinafter, referred to as “specification result”) that specifies any of “safety”, “cautionary travel”, and “running impossible” and position information of the specified step or groove 92. Output to the processing unit 40.

The image processing unit 40 inputs the bird's-eye view image 78 from the generation unit 34, and also inputs the identification result and the position information from the road surface shape determination unit 48. When the specific result is “caution driving” or “running impossible”, that is, when it is determined that the road surface shape requiring attention exists, the video processing unit 40 superimposes an image showing the road surface shape requiring attention on the overhead view video 78. Let FIG. 11 shows a bird's-eye view image 78 generated by the image processing unit 40. This corresponds to the case where the specific result is "caution driving". The video processing unit 40 generates the cautionary traveling notification image 82 to be shown in a part of the bird's-eye view video 78 and including the position information. The caution driving notification image 82 is an image for notifying the “caution driving”. The video processing unit 40 superimposes the cautionary traveling notification image 82 on the area in the overhead view video 78. Here, the cautionary traveling notification image 82 is arranged on the right side of the vehicle icon 80.

FIG. 12 shows another bird's-eye view image 78 generated by the image processing unit 40. This corresponds to the case where the specific result is "running impossible". The video processing unit 40 generates the travel-prohibition notification image 84 to be shown in a part of the bird's-eye view video 78 and including the position information. The travel-prohibition notification image 84 is an image for notifying that “travel is impossible”. The video processing unit 40 superimposes the travel-prohibition notification image 84 on the area in the overhead view video 78. Here, the travel-prohibition notification image 84 is arranged on the right side of the vehicle icon 80. Return to FIG. The video processing unit 40 causes the display control unit 42 to display the bird's-eye view image 78 on which the cautionary traveling notification image 82 or the travel-prohibited notification image 84 is superimposed, or the bird's-eye view image 78 on which the cautionary traveling notification image 82 and the travel-prohibited notification image 84 are not superimposed. Output.

The display control unit 42 inputs the bird's-eye view image 78 from the image processing unit 40. The display control unit 42 causes the display unit 50 to display the bird's-eye view video 78 by executing the process of displaying the bird's-eye view video 78. The display unit 50 is, for example, a display panel. When the caution notice image 82 or the prohibition notice image 84 is superimposed on the bird's-eye view image 78, the bird's-eye view image 78 as shown in FIG. 11 or 12 is displayed on the display unit 50.

Various examples of the bird's-eye view video 78 on which the image is superimposed in the video processing unit 40 will be described below. FIG. 13 shows still another bird's-eye view image 78 generated by the image processing unit 40. Here, the vehicle 100 is moving forward. In the overhead view video 78, a left front notification image 150, a front notification image 152, a right front notification image 154, a left rear notification image 156, a rear notification image 158, and a right rear notification image 160 are shown. Here, the left front notification image 150, the front notification image 152, and the right front notification image 154 are arranged in front of the own vehicle icon 80, and the left rear notification image 156, the rear notification image 158, and the right rear notification image 160 are It is arranged behind the car icon 80.

The left front notification image 150 is an image for indicating whether the detection result of the left front sensor 120 is “no step”, “stepped”, or “cliff”. Here, “with step” corresponds to “cannot travel” described above, “without step” corresponds to “careful driving” or “safety” described above, and “cliff” means “with step” described above. In the case of “”, it corresponds to the case where the height h is “30 cm” or more. The “cliff” also includes the case where the sensor 110 is not operating.

The front notification image 152, the right front notification image 154, the left rear notification image 156, the rear notification image 158, and the right rear notification image 160 are the front sensor 112, the right front sensor 122, the left rear sensor 124, the rear sensor 116, and the right rear sensor. Also shown for 126. Here, as an example, the left front notification image 150 indicates “no step”, the front notification image 152 indicates “stepped”, and the right front notification image 154 to the right rear notification image 160 indicate “cliff”. When the vehicle 100 is moving forward, if the left rear sensor 124, the rear sensor 116, and the right rear sensor 126 for detecting the rear are not operating, the left rear notification image 156 to the right rear notification image 160 are also “cliff”. Indicates.

FIG. 14 shows still another bird's-eye view image 78 generated by the image processing unit 40. For example, when the speed of the vehicle 100 is 4 to 5 km/h or less, the bird's-eye view image 78 is arranged with the left side notification image 162 and the right side notification image 164 instead of the left front notification image 150 and the like in FIG. 13. It Here, the left side notification image 162 is arranged on the left side of the own vehicle icon 80, and the right side notification image 164 is arranged on the right side of the own vehicle icon 80. The left side notification image 162 is an image for indicating whether the detection result of the left side sensor 114 is “no step”, “stepped”, or “cliff”. The right side notification image 164 is similarly shown for the right side sensor 118. Here, as an example, the left side notification image 162 indicates “cliff”, and the right side notification image 164 indicates “stepped”.

FIG. 15 shows still another bird's-eye view image 78 generated by the image processing unit 40. This corresponds to the case where the right front notification image 160, the left side notification image 162, and the right side notification image 164 are displayed from the left front notification image 150. This is displayed in the same situation as in FIG.

16A and 16B show images generated by the image processing unit 40. Until now, the video processing unit 40 has superimposed an image showing a road surface shape that requires attention on the overhead view video 78. On the other hand, here, the image showing the road surface shape requiring attention is superimposed on the video imaged by the imaging unit 10. In FIG. 16A, the left front notification image 150, the front notification image 152, and the right front notification image 154 are arranged in the front video 170 captured by the front imaging unit 12. The left front notification image 150, the front notification image 152, and the right front notification image 154 are the same as before. In FIG. 16B, the rear notification image 158 is arranged in the rear image 172 captured by the rear image capturing unit 16. The rear notification image 158 is the same as before.

In terms of hardware, this configuration can be realized by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it can be realized by a program loaded in the memory, but here it is realized by cooperation of them. It depicts the functional blocks that will be used. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.

The operation of the vehicular display device 30 having the above configuration will be described. FIG. 17 is a flowchart showing a display procedure by the vehicle display device 30. The generation unit 34 generates the bird's-eye view image 78 (S10). When the vehicle is traveling or stopped at a speed lower than the predetermined speed (Y in S12), the road surface shape acquisition unit 46 acquires the road surface shape (S14). When there is a road surface shape that requires attention (Y in S16), the video processing unit 40 superimposes the image on the overhead video 78 (S18). The display unit 50 displays the bird's-eye view image 78 (S20). If the vehicle is not running or stopped at a speed lower than the predetermined speed (N in S12) or there is no road surface shape requiring attention (N in S16), the process is skipped to step 20.

According to the present embodiment, when it is determined that the road surface shape requiring attention exists, the image showing the road surface shape requiring attention is superimposed on the bird's-eye view image, so that the road surface shape requiring attention can be notified. Further, since the road surface shape requiring attention is notified, it is possible to grasp the presence/absence of a step even in the overhead image. Further, since the road surface shape is acquired by acquiring the distance from the vehicle body of the vehicle to the road surface within the range where the road surface shape can be acquired, the distance measured by the sensor can be used.

Further, when the acquired traveling condition is traveling or stopping at a speed lower than a predetermined speed, the road surface shape with respect to the periphery of the vehicle is acquired, so that a step or a cliff existing at a short distance from the vehicle can be notified. Further, since it is determined whether or not there is a road surface shape having a height difference that makes it difficult for the vehicle to travel as a road surface shape requiring attention, it is possible to detect a step that is difficult to be shown in the overhead view image. In addition, since it is determined whether or not there is a step shape that becomes lower as the vehicle moves away from the vehicle, it is possible to notify the existence of a step that is difficult to be shown particularly in the overhead view image.

(Example 2)
Next, a second embodiment will be described. The second embodiment, like the first embodiment, relates to a vehicular display device that displays a bird's-eye view image showing the presence/absence of a step. In the second embodiment, particularly, the case where there is a road surface shape having a height difference that requires attention when the vehicle door is opened will be described. The vehicle 100 according to the second embodiment is of the same type as in FIGS. 1 and 2. Here, the difference from the above will be mainly described.

FIG. 18 shows the configuration of the vehicular display device 30 according to the second embodiment of the present invention. In the vehicle display device 30, the seating information acquisition unit 52 is added to the vehicle display device 30 of FIG. The seating information acquisition unit 52 is connected to a seating sensor (not shown) arranged in each of a plurality of seats in the vehicle 100, and receives a detection result from each seating sensor. A known technique may be used for the seating sensor. The detection result from the seating sensor indicates whether or not an occupant is sitting in the seat, and can be said to be seating presence/absence information. Based on such seating presence/absence information, the seating information acquisition unit 52 identifies the seat on which the occupant is seated. The seating information acquisition unit 52 outputs information regarding the seat on which the occupant is seated to the road surface shape determination unit 48.

The road surface shape acquisition unit 46 acquires data indicating the road surface shape around the vehicle 100 by analyzing the detection result from the sensor 110, as in the past. However, the road surface shape, which is the premise of the description, is different from the conventional one. FIG. 19 shows an outline of processing in the road surface shape acquisition unit 46. Here, there is a high step portion 96 that is “h” higher than the road surface 90. Further, the lower end line 132 when the door is opened and closed indicates a line through which the lower end of the door passes when the door of the vehicle 100 is opened and closed.

As shown, a distance L1"' for the point P1, a distance L2"' for the point P2, a distance L3"' for the point P3, and a distance L4"' for the point P4 are shown. Therefore, in the data indicating the road surface shape, a combination of the point P1 and the distance L1′″, a combination of the point P2 and the distance L2′″, a combination of the point P3 and the distance L3″′, and a point P4 and the distance are included. Combinations with L4''' are included. Such data shows a road surface shape as shown in FIG. Returning to FIG. When the road surface shape acquisition unit 46 acquires the data indicating the road surface shape, the road surface shape acquisition unit 46 outputs the data indicating the road surface shape to the road surface shape determination unit 48.

The road surface shape determination unit 48 inputs data indicating the road surface shape from the road surface shape acquisition unit 46. Further, the road surface shape determination unit 48 inputs information regarding the seat on which the occupant is seated from the seating information acquisition unit 52. Based on the road surface shape indicated in the data input from the road surface shape acquisition unit 46, the road surface shape determination unit 48 sets a road surface shape that requires attention as a road surface having a height difference that makes it difficult to open the door of the vehicle 100. Determine if a shape exists. In particular, the road surface shape determination unit 48 determines whether or not there is a road surface shape with a height difference that makes it difficult to open the seated door acquired by the seating information acquisition unit 52 among the doors of the vehicle 100. .. As in the past, in order to determine whether or not there is a road surface shape requiring attention, the road surface shape determination unit 48 stores a reference pattern of the road surface shape requiring attention in the road surface shape as a table.

FIG. 20 shows a data structure of a table stored in the road surface shape determination unit 48. When the road surface shape is an upward step, the height h is used as a parameter. Further, “safety”, “door opening caution”, and “door opening impossible” are defined according to the value of the parameter. Here, the case where the value corresponding to the value obtained by subtracting about 5 to 10 cm from the lower end line 132 at the time of opening and closing the door with respect to the ground height when an adult (eg, weight 70 to 100 kg) is seated on the seat corresponding to the door is "door. It is equivalent to "Open caution". Returning to FIG.

The road surface shape determination unit 48 refers to the table shown in FIG. 20, and based on the road surface shape shown in the data input from the road surface shape acquisition unit 46, “safety” is applied to the door close to the seat on which the occupant is seated. , "Do not open the door", or "Do not open the door" is specified. The road surface shape determination unit 48 informs the image processing unit 40 of information (hereinafter, referred to as “identification result”) that specifies any one of “safety”, “door opening caution”, and “door not open”, and the door information. Output.

The image processing unit 40 inputs the bird's-eye view image 78 from the generation unit 34, and inputs the identification result and the door information from the road surface shape determination unit 48. When the specific result is “door open caution” or “door open impossible”, that is, when it is determined that there is a road surface shape requiring attention, the image processing unit 40 causes the bird's-eye view image 78 of an image showing the road surface shape requiring attention. Superimpose on. FIG. 21 shows a bird's-eye view image 78 generated by the image processing unit 40. This corresponds to the case where the identification result is “door open caution” and the right door of the vehicle 100 is indicated in the identified door information. The video processing unit 40 generates the door opening caution notification image 86 to be shown in the area overlapping the right door of the vehicle icon 80 in the overhead view video 78. The door open caution notification image 86 is an image for notifying the "door open caution". The video processing unit 40 superimposes the door opening caution notification image 86 on the door on the right side of the vehicle icon 80 in the overhead view video 78. When the right and front doors are shown in the door information, the video processing unit 40 may superimpose the door opening caution image 86 only on the right and front doors of the vehicle icon 80.

FIG. 22 shows another bird's-eye view image 78 generated by the image processing unit 40. This corresponds to the case where the specific result is "door cannot be opened". The video processing unit 40 generates the door opening prohibition notification image 88 to be shown in the area overlapping the right side door of the vehicle icon 80 in the overhead view video 78. The door open prohibition notification image 88 is an image for notifying that "the door cannot be opened". The video processing unit 40 superimposes the door open prohibition notification image 88 on the door on the right side of the vehicle icon 80 in the overhead view video 78. When the right and front doors are shown in the door information, the video processing unit 40 may superimpose the door opening failure notification image 88 only on the right and front doors of the vehicle icon 80. Returning to FIG. The image processing unit 40 displays the bird's-eye view image 78 on which the door open caution notification image 86 or the door open prohibition notification image 88 is superimposed, or the bird's eye view video 78 on which the door open caution notice image 86 and the door open prohibition notification image 88 are not superimposed. Output to the control unit 42.

According to the present embodiment, it is determined whether or not there is a road surface shape having a height difference that makes it difficult to open the door of the vehicle. Therefore, it is possible to notify the road surface shape that is difficult to show in the overhead image. Further, since the road surface shape having a height difference that makes it difficult to open the door of the vehicle is superimposed as an image, the possibility of hitting the door on a step can be reduced. Further, since it is determined whether or not there is a road surface shape having a height difference that makes it difficult to open a seated door, an image can be displayed on a door that is likely to be opened. Further, since the image is displayed on the door that is likely to be opened, it is possible to prevent the situation where the area of the image to be superimposed on the overhead view video becomes too large.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that this embodiment is an exemplification, and that various modifications can be made to the combinations of the respective constituent elements and the respective processing processes, and that such modifications are within the scope of the present invention. ..

The combination of Examples 1 and 2 is also effective. According to this modification, the effect obtained by combining the first and second embodiments can be obtained.

10 image pickup unit, 12 front image pickup unit, 14 left side image pickup unit, 16 rear image pickup unit, 18 right side image pickup unit, 30 vehicle display device, 32 image acquisition unit, 34 generation unit, 40 image processing unit, 42 display control unit , 44 running condition acquisition unit, 46 road surface shape acquisition unit, 48 road surface shape determination unit, 50 display unit, 100 vehicle, 110 sensor, 112 front sensor, 114 left side sensor, 116 rear sensor, 118 right side sensor, 120 left front Sensor, 122 right front sensor, 124 left rear sensor, 126 right rear sensor.

Claims (9)

An image acquisition unit that acquires an image of the surroundings of the vehicle,
A generation unit that generates a bird's-eye view image by converting the viewpoint as seen from above the vehicle with respect to the image acquired by the image acquisition unit;
A traveling condition acquisition unit for acquiring the traveling condition of the vehicle,
A road surface shape acquisition unit that acquires a road surface shape in the vicinity of the vehicle when the traveling condition acquired by the traveling condition acquisition unit is traveling or stopped at a speed lower than a predetermined speed ,
Based on the road surface shape acquired by the road surface shape acquisition unit, a road surface shape determination unit that determines whether or not there is a road surface shape that requires attention,
When it is determined that the road surface shape requiring attention is present in the road surface shape determining unit, a video processing unit that superimposes an image showing the road surface shape requiring attention on the overhead view video,
A display control unit that causes the display unit to display a bird's-eye view image on which images are superimposed in the video processing unit;
A vehicle display device comprising: An image acquisition unit that acquires an image of the surroundings of the vehicle,
A generation unit that generates a bird's-eye view image by converting the viewpoint as seen from above the vehicle with respect to the image acquired by the image acquisition unit;
A road surface shape acquisition unit that acquires a road surface shape in the vicinity of the vehicle,
Based on the road surface shape acquired by the road surface shape acquisition unit, as a road surface shape that requires attention , a road surface shape determination unit that determines whether or not there is a road surface shape with a height difference that makes traveling of the vehicle difficult. ,
When it is determined that the road surface shape requiring attention is present in the road surface shape determining unit, a video processing unit that superimposes an image showing the road surface shape requiring attention on the overhead view video,
A display control unit that causes the display unit to display a bird's-eye view image on which images are superimposed in the video processing unit;
A vehicle display device comprising: The road shape determination unit, a road shape travel a height difference to be difficult of the vehicle, to claim 2, wherein the stepped shape decreases away from the vehicle to determine whether there The display device for a vehicle described. An image acquisition unit that acquires an image of the surroundings of the vehicle,
A generation unit that generates a bird's-eye view image by converting the viewpoint as seen from above the vehicle with respect to the image acquired by the image acquisition unit;
A road surface shape acquisition unit that acquires a road surface shape in the vicinity of the vehicle,
Based on the road surface shape acquired by the road surface shape acquisition unit , a road surface shape that determines whether or not there is a road surface shape with a height difference that makes it difficult to open the door of the vehicle as a road surface shape that requires attention A judgment unit,
When it is determined that the road surface shape requiring attention is present in the road surface shape determining unit, a video processing unit that superimposes an image showing the road surface shape requiring attention on the overhead view video,
A display control unit that causes the display unit to display a bird's-eye view image on which images are superimposed in the video processing unit;
A vehicle display device comprising: Further comprising a seating information acquisition unit for acquiring seating presence/absence information for a plurality of seats in the vehicle,
The road surface shape determination unit determines whether or not there is a road surface shape with a height difference that makes it difficult to open the seated door acquired by the seating information acquisition unit among the vehicle doors. The display device for a vehicle according to claim 4 , wherein the display device for a vehicle is used. The road shape obtaining unit, the acquisition possible range of the road surface shape, any one of claims 1-5, characterized in that to obtain the road surface shape by obtaining the distance from the vehicle body of the vehicle to the road surface The display device for a vehicle according to item 1. Acquiring a video image of the surroundings of the vehicle,
A step of generating a bird's-eye view image by converting the viewpoint as seen from above the vehicle with respect to the acquired image;
Acquiring the traveling status of the vehicle,
A step of acquiring a road surface shape in the vicinity of the vehicle when the acquired traveling condition is traveling or stopped at a speed lower than a predetermined speed ;
Based on the acquired road surface shape, a step of determining whether or not there is a road surface shape requiring attention,
If it is determined that there is a road surface shape requiring attention, a step of superimposing an image showing the road surface shape requiring attention on the overhead image,
A step of displaying a bird's-eye view image on which the image is superimposed on the display unit,
A vehicle display method comprising: Acquiring a video image of the surroundings of the vehicle,
A step of generating a bird's-eye view image by converting the viewpoint as seen from above the vehicle with respect to the acquired image;
Acquiring a road surface shape around the vehicle,
Based on the obtained road surface shape, as a road surface shape requiring attention, a step of determining whether or not there is a road surface shape having a height difference that makes traveling of the vehicle difficult ,
If it is determined that there is a road surface shape requiring attention, a step of superimposing an image showing the road surface shape requiring attention on the overhead image,
A step of displaying a bird's-eye view image on which the image is superimposed on the display unit,
A vehicle display method comprising: Acquiring a video image of the surroundings of the vehicle,
A step of generating a bird's-eye view image by converting the viewpoint as seen from above the vehicle with respect to the acquired image;
Acquiring a road surface shape around the vehicle,
Based on the acquired road surface shape, as a road surface shape requiring attention, a step of determining whether or not there is a road surface shape having a height difference that makes it difficult to open the door of the vehicle ,
If it is determined that there is a road surface shape requiring attention, a step of superimposing an image showing the road surface shape requiring attention on the overhead image,
A step of displaying a bird's-eye view image on which the image is superimposed on the display unit,
A vehicle display method comprising:

Images