WO2020139582A1

WO2020139582A1 - Surround view by drones

Info

Publication number: WO2020139582A1
Application number: PCT/US2019/066166
Authority: WO
Inventors: Bastian Zydek; Dominik Froehlich
Original assignee: Continental Automotive Systems Inc
Current assignee: Aumovio Systems Inc
Priority date: 2018-12-27
Filing date: 2019-12-13
Publication date: 2020-07-02
Anticipated expiration: 2021-06-27
Also published as: US20200207273A1; US11577647B2

Abstract

An apparatus includes a visual display to be viewed by a vehicle occupant. At least one drone includes a camera. A controller is configured to receive images from the camera on the at least one drone and generate an overhead view of the vehicle based on the images received from the at least one drone and display the overhead view on the visual display.

Description

SURROUND VIEW BY DRONES

BACKGROUND

[0001] A vehicle can include a plurality of cameras that generate images of vehicle position that are combined together to present the driver with an overhead view of a position of the vehicle in relation to a surrounding environment. This type of visual display can assist the driver when maneuvering in tight constraints. In certain situations, the coverage provided by the cameras may be insufficient to fully understand the position of the vehicle in relation to the surrounding environment.

SUMMARY

[0002] In one exemplary embodiment, an apparatus includes a visual display to be viewed by a vehicle occupant. At least one drone includes a camera. A controller is configured to receive images from the camera on the at least one drone and generate an overhead view of the vehicle based on the images received from the at least one drone and display the overhead view on the visual display.

[0003] In a further embodiment of any of the above, the vehicle is coupled to a trailer and the overhead view shows both the vehicle and the trailer.

[0004] In a further embodiment of any of the above, the vehicle is provided with at least one camera to generate vehicle position data for the overhead view of the vehicle. The at least one drone is used to extend a visualization area to include position data for the trailer.

[0005] In a further embodiment of any of the above, the at least one drone is deployed from the vehicle, the trailer, or nearby infrastructure. [0006] In a further embodiment of any of the above, the visual display displays images generated by at least one camera associated with the vehicle.

[0007] In a further embodiment of any of the above, the at least one camera includes a plurality of camera.

[0008] In a further embodiment of any of the above, the overhead view includes a 360 degree surround overhead view of the vehicle.

[0009] In a further embodiment of any of the above, the at least one drone is maneuvered by the vehicle occupant via HMI.

[0010] In another exemplary embodiment, a method of providing an overhead view of a vehicle includes (a) deploying at least one drone in response to the vehicle on a drivable surface encountering a visual obstruction relative to the vehicle, (b) generating an overhead view of the vehicle based at least on images from the at least one drone and (c) displaying the overhead view on a visual display.

[0011] In a further embodiment of any of the above, step (a) includes deploying the at least one drone from a docking station on at least one of the vehicle, a trailer coupled to the vehicle, or nearby infrastructure.

[0012] In a further embodiment of any of the above, step (b) includes generating the overhead view of the vehicle based on images from the at least one drone and images from at least one camera associated with the vehicle. [0013] In a further embodiment of any of the above, the at least one drone extends a field of view surrounding the vehicle beyond a field of view of the at least one camera on the vehicle.

[0014] In a further embodiment of any of the above, the visual obstruction relative to vehicle includes reversing the vehicle along the drivable surface.

[0015] In a further embodiment of any of the above, the visual obstruction relative to the vehicle includes a structure adjacent the drivable surface.

[0016] In a further embodiment of any of the above, step (a) includes maneuvering the drone based on a position of the vehicle relative to the drivable surface.

[0017] In a further embodiment of any of the above, the position of the vehicle relative to the drivable surface includes the vehicle at an intersection of multiple drivable surfaces.

[0018] In a further embodiment of any of the above, the vehicle is coupled to a trailer.

[0019] In a further embodiment of any of the above, step (b) includes modifying the overhead view of the vehicle based on input from a vehicle occupant.

[0020] In a further embodiment of any of the above, the at least one drone is maneuvered via HMI to modify the overhead view of the vehicle.

[0021] In a further embodiment of any of the above, the overhead view includes a 360 degree surround exterior view of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The disclosure can be further understood by reference to the following detailed description when considered in connection with the accompanying drawings wherein:

[0023] Figure 1 is a schematic illustration of a vehicle, drone, and visualization system incorporating the subject disclosure.

[0024] Figure 2 is an example of an overhead view of vehicle position.

[0025] Figure 3 is a schematic overhead illustration of the vehicle, drone, and visualization system incorporating the subject disclosure.

[0026] Figure 4 illustrates a flowchart of a method of providing an overhead surround view according to the subject disclosure.

[0027] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, unless such features are incompatible.

DETAILED DESCRIPTION

[0028] The subject disclosure comprises a visualization system that provides for an improved overhead view of vehicle position for a vehicle occupant. In one example shown in

Figure 1, a vehicle 10 is pulling a trailer 12 along a drivable surface 14, such as a road, driveway, parking lot, or loading dock. The vehicle 10 includes a visual display 16 that is to be viewed by the vehicle occupant. The visual display 16, among other things, is used to show an overhead view of the vehicle 10 in relation to a surrounding environment (see Figure 2, for example). In the illustrated example, the overhead view is a 360 degree exterior surround view of the vehicle 10. The visual display 16 can include one of an integrated display in the vehicle 10, such as part of an entertainment system. Alternatively, the visual display could include an external display, or a personal mobile device, such as a cell phone, tablet, or laptop.

[0029] At least one drone 18 is used to communicate vehicle position data to the visual display 16. The data may first be communicated to a system controller 20, which analyzes the data and then communicates the data to the visual display 16, through the use of a transmitter/receiver 19 on the drone 18 and a transmitter/receiver 21 on the controller 20. In one example, the position data includes images captured by a camera 17 on the drone 18. The position data from the drone 18 can be used to show both the position of the vehicle 10 and the trailer 12 in the overhead view.

[0030] In another example, the vehicle 10 is provided with one or more cameras 22 that send data to the controller 20 to generate the vehicle position for the overhead view of the vehicle 10. In the illustrated example, the cameras 22 are associated with the vehicle 10. The controller 20 combines the various images captured by the cameras 22 to generate the overhead view as known. The drone 18 can then be used as needed to enlarge or extend a visualization area to include position data for the trailer 12. Additionally, the cameras 17 on the drone 18 can be used to provide images of areas that are outside a field of view of the cameras 22 on the vehicle 10.

[0031] The drone 18 can be selectively or automatically deployed from the vehicle 10, the trailer 12, or a nearby infrastructure 24 such as a building, parking structure, loading dock, etc. In one example, the vehicle occupant maneuvers the drone 18 via Human Machine Interface (HMI), e.g. touch-screen, knob, etc.

[0032] As such, the system provides an improved overview view for a driver maneuvering a vehicle with or without a trailer by having a drone 18 fly over a scenario, e.g. vehicle and/or trailer, and displaying the scenario in a bird s eye perspective to the driver via surround view visualization. In one example, the surround view image provided by the drone 18 is by default controlled by the position of the vehicle 10 and/or trailer 12. The HMI can then be selectively used to adjust the image by using zoom a d shift commands

[0033] Traditional surround view systems use image data coming from the cameras 22 on the vehicle 10 to stitch together a compiled image view through which the driver gets a displayed visualization of the vehicle with any surrounding obstacles from a bird's eye perspective. As discussed above, this helps the driver to visualize the surrounding vehicles in relation to the driver’s vehicle and eases maneuvering of the vehicle in tight spaces and parking situations. In certain situations, the driver can choose where the viewpoint of the visualization is to be set.

[0034] The subject invention uses the footage from the drone 18 either to enlarge/extend the visualization in a situation where a trailer 12 is hitched to the vehicle 10, or to provide the surround view visualization of the vehicle 10 itself. This is accomplished by the drone 18 hovering above the vehicle or by flying along the vehicle’s path. The drone 18 can be selectively or automatically engaged from the vehicle 10, the trailer 12, or the infrastructure 24. A drone-dock 26 can be mounted to the trailer 12, vehicle 10, or the infrastructure 12 to hold the drone 18 when the drone 18 is not deployed. The drone-dock 26 can include charging capability, communication interfaces, etc. as needed for the operation of the drone 18. [0035] This type of visualization system significantly improves driver maneuvering for vehicles 10 with trailers 12. The drone 18 could also be utilized to capture critical scenarios like driveways where there are obstructions and drivers cannot see any lateral traffic or obstacles. Or, the drone 18 could be used in loading dock scenarios where drivers of a truck have to perfectly align the truck/trailer with a loading dock. The drone 18 could also automatically engage in such scenarios and the footage of the drone 18 could be merged with the surround view footage from the cameras 22 on the vehicle 10.

[0036] In one example, the drone 18 is maneuvered by the vehicle occupant, e.g. driver, pinching and zooming and scrolling/dragging a surface of a touchscreen into certain directions on the surround view visualization. An algorithm translates this human machine input into a certain drone movement. Input can also be given by any other HMI input such as a 3D input-knob (e.g. iDrive controller) for example.

[0037] Figure 3 is a schematic view of the vehicle 10, drone 18, and visualization system incorporating the subject disclosure. In the illustrated example, the vehicle 10 is located at an intersection 36 of drivable surfaces 14, such as a driveway entering onto a road. When the vehicle 10 is located on a driveway 14 with the intention of pulling out onto the road 14 with traffic traveling in multiple directions (see vehicles 10B and IOC), it is possible that a view of road 14 with multiple direction of traffic may be partially obstructed.

[0038] In this example, the view of the vehicle occupant and cameras 22 on the vehicle

10 is obstructed by a structure 30, such as a building or a tree. The dashed lines 32 represents a field of view of the cameras 22 that is obstructed by the structure 30. Therefore, when the drone

18 is deployed, an overhead surround view that is displayed on the visual display 16 can be expanded beyond the structure 30 and the trailer 12 as shown by dashed line 34. Therefore, the overhead surround view generated at least in part from images from the drone 18 expands the field of view of the overhead surround view generated for the visual display 16. This can warn the vehicle occupant of oncoming traffic, such as vehicle 10B, or provide a larger view of the vehicle 10 and trailer 12 on the drivable surface 14.

[0039] Figure 4 illustrates an example method 100 of providing an overhead surround view of the vehicle 10 with or without the trailer 12. The method 100 includes deploying at least one drone 18 when the vehicle 10 encountering a visual obstruction relative to the drivable surface 14. (Step 102). The visual obstruction can occur in a number of circumstances, such as reversing the vehicle 10 or a structure 30 that obstructs a field of view of the vehicle occupant or one of the cameras 22 on the vehicle 10. When the drone 18 is deployed, the drone 18 departs from the drone- dock 26 on at least one of the vehicle 10, the trailer 12 coupled to the vehicle 10, or the infrastructure 24.

[0040] The drone 18 can be maneuvered based on a number of criteria. In one example, the drone 18 is maneuvered based on a position of the vehicle 10 on the drivable surface 14. The position of the vehicle 10 may include the vehicle 10 located at the intersection 36 of drivable surfaces 14, such as exiting a narrow driveway onto a wider road, or at a loading dock.

[0041] The controller 20 can then generate an overhead view of the driveable surface

14 based on the images received from the drone 18. (Step 104). The overhead view of the driveable surface 14 based on the images from the drone 18 can provide a wider field of view than just the cameras 22 associated with the vehicle 10 or provide a view of the area surrounding the vehicle

10 that is obstructed from the cameras 22. Additionally, the controller 20 can generate the overhead surround view based on both the images received from the drone 18 and the images received by the cameras 22. Therefore, the images from the drone 18 can be used to supplement or extend the field of view captured by the images from the cameras 22 associated with the vehicle 10 or vice versa.

[0042] Furthermore, when the overhead surround view is displayed on the visual display 16 (Step 106), the vehicle occupant can modify the overhead view further by interacting with the HMI. For example, the vehicle occupant could zoom in or out of the overhead view, change a direction of the camera 17 on the drone 18 to provide another view, and/or maneuver the drone 18 to provide another view.

[0043] It should also be understood that although a particular component arrangement is disclosed in the illustrated embodiment, other arrangements will benefit herefrom. Although particular step sequences are shown, described, and claimed, it should be understood that steps may be performed in any order, separated or combined unless otherwise indicated and will still benefit from the present invention.

[0044] Although the different examples have specific components shown in the illustrations, embodiments of this invention are not limited to those particular combinations. It is possible to use some of the components or features from one of the examples in combination with features or components from another one of the examples.

[0045] Although an example embodiment has been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. For that reason, the following claims should be studied to determine their true scope and content.

Claims

CLAIMS What is claimed is:

1. An apparatus comprising:

a visual display to be viewed by a vehicle occupant;

at least one drone including a camera; and

a controller configured to receive images from the camera on the at least one drone and generate an overhead view of the vehicle based on the images received from the at least one drone and display the overhead view on the visual display.

2. The apparatus according to claim 1, wherein the vehicle is coupled to a trailer and the overhead view shows both the vehicle and the trailer.

3. The apparatus according to claim 2, including providing the vehicle with at least one camera to generate vehicle position data for the overhead view of the vehicle, and wherein the at least one drone is used to extend a visualization area to include position data for the trailer.

4. The apparatus according to claim 3, wherein the at least one drone is deployed from the vehicle, the trailer, or nearby infrastructure.

5. The apparatus according to claim 1, wherein the visual display displays images generated by at least one camera associated with the vehicle and the visual display includes one of an integrated vehicle display, an external display, or a mobile device.

6. The apparatus according to claim 5, wherein the at least one camera includes a plurality of camera.

7. The apparatus according to claim 1, wherein the overhead view includes a 360 degree surround overhead view of the vehicle.

8 The apparatus according to claim 1, wherein the at least one drone is maneuvered by the vehicle occupant via HMI.

9. A method of providing an overhead view of a vehicle comprising the steps of:

(a) deploying at least one drone in response to the vehicle on a drivable surface encountering a visual obstruction relative to the vehicle;

(b) generating an overhead view of the vehicle based at least on images from the at least one drone; and

(c) displaying the overhead view on a visual display.

10. The method according to claim 9, wherein step (a) includes deploying the at least one drone from a docking station on at least one of the vehicle, a trailer coupled to the vehicle, or nearby infrastructure.

11. The method according to claim 9, wherein step (b) includes generating the overhead view of the vehicle based on images from the at least one drone and images from at least one camera associated with the vehicle.

12. The method according to claim 11, wherein the at least one drone extends a field of view surrounding the vehicle beyond a field of view of the at least one camera on the vehicle.

13. The method according to claim 9, wherein the visual obstruction relative to vehicle includes reversing the vehicle along the drivable surface.

14. The method according to claim 9, wherein the visual obstruction relative to the vehicle includes a structure adjacent the drivable surface.

15. The method according to claim 14, wherein step (a) includes maneuvering the drone based on a position of the vehicle relative to the drivable surface.

16. The method according to claim 15, wherein the position of the vehicle relative to the drivable surface includes the vehicle at an intersection of multiple drivable surfaces.

17. The method according to claim 16, wherein the vehicle is coupled to a trailer.

18. The method according to claim 9, wherein step (b) includes modifying the overhead view of the vehicle based on input from a vehicle occupant.

19. The method according to claim 18, including maneuvering the at least one drone via HMI to modify the overhead view of the vehicle.

20. The method according to claim 9, wherein the overhead view includes a 360 degree surround exterior view of the vehicle.