GB2626806A

GB2626806A - A method for generating a three-dimensional model of the surroundings of a motor vehicle by an assistance system, a computer program product, a non-transitory

Info

Publication number: GB2626806A
Application number: GB2301654.6A
Authority: GB
Inventors: Monninger Thomas
Original assignee: Mercedes Benz Group AG
Current assignee: Mercedes Benz Group AG
Priority date: 2023-02-06
Filing date: 2023-02-06
Publication date: 2024-08-07

Abstract

A three-dimensional model (14) of the surroundings (16) of a motor vehicle (10) is generated by a system (12) by capturing at least a first image (28) of the surroundings (16) in a first time step (t1) and a second image (30) of the surroundings (16) in a second time step (t2). The second time step (t2) is after the first time step (t1), and the second image (30) shows the surroundings (16) from a different perspective than the first image (28). At least the first image (28) and the second image (30) are rendered to the three-dimensional model (14) by using a neural radiance field method (NeRF). The position and/or orientation of the vehicle may be taken into account when generating the model. The system may be employed for drivers/users to visualise surroundings during bad weather (obscured vision), gain an impression of a route before setting off, or annotate the visualisation of the environment to present information.

Description

A method for generating a three-dimensional model of the surroundings of a motor vehicle by an assistance system, a computer program product, a non-transitory computer-readable storage medium, as well as a corresponding assistance system

FIELD OF THE INVENTION

[0001] The present invention relates to the field of automobiles. More specifically, the present invention relates to a method for generating a three-dimensional model of the surroundings of a motor vehicle by an assistance system of the motor vehicle. Furthermore, the present invention relates to a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as to a corresponding assistance system.

BACKGROUND INFORMATION

[0002] Human drivers in some situations require a visualization of their surrounding environment, which for example may serve as additional guidance in bad weather conditions, may be used to gain an impression of the route before setting off, or may be used to display environmental information to the driver by annotating the visualization of the environment.

SUMMARY OF THE INVENTION

[0003] It is an object of the present invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding assistance system, by which a visualization of the surroundings of a motor vehicle can be generated in a more efficient way.

[0004] This object is solved by a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding assistance system.

[0005] One aspect of the invention relates to a method for generating a three-dimensional model of the surroundings of a motor vehicle by an assistance system of the motor vehicle. At least a first image of the surroundings in a first time step and a second image of the surroundings in a second time step is captured by a capturing device of the assistance system, wherein the second time step is after the first time step, and wherein the second image shows the surroundings from a different perspective than the first image. At least the first image and the second image are rendered into the three-dimensional model by using a neural radiance field method by an electronic computing device of the assistance system.

[0006] Therefore, a method is provided by which the scene reconstruction allows to visualize an environment to the driver. In particular, in bad weather or sight conditions the visualization is a visual aid for the driver. Furthermore, the visualization may serve as basis to provide further environmental information such as annotation, for example display navigation, or car-to-x-events. The visualization may be conditioned to serve as a theme or gimmick for the driver. The visualization may be provided before starting to drive in order to give a driver the chance to drive the route virtually. An on-board solution provides scene reconstruction for a driven route, and maximizes privacy, for example the scene reconstruction may be performed also for private ground, since data stays in car. Furthermore, the backend solution allows to share three-dimensional models with other vehicles to provide the functionality on roads that were not driven by the vehicle before. Another advantage is that the three-dimensional scene reconstruction works on the basis of a very small set of images and generates highly compressed models. Therefore, it is a very data-efficient process.

[0007] According to an embodiment a position and/or an orientation of the motor vehicle during capturing the images is taken into consideration for generating the three-dimensional model.

[0008] According to another embodiment the three-dimensional model is transmitted to an external electronic computing device.

[0009] In another embodiment the three-dimensional model is used for generating a further three-dimensional model for further surroundings by the external electronic computing device.

[0010] In another embodiment the three-dimensional model and/or the further three-dimensional model is provided for further motor vehicles.

[0011] According to another embodiment the three-dimensional model is displayed on a display device of the assistance system.

[0012] In particular, the provided method is a computer-implemented method. Therefore, another aspect of the invention relates to a computer program product comprising program codes means for performing a method according to the preceding aspect.

[0013] A still further aspect of the invention relates to a non-transitory computer-readable storage medium comprising at least the computer program product according to the preceding aspect.

[0014] A still further aspect of the invention relates to an assistance system for generating a three-dimensional model of the surroundings of a motor vehicle, comprising at least one capturing device and one electronic computing device, wherein the assistance system is configured for performing a method according to the preceding aspect. In particular, the method is performed by the assistance system.

[0015] The electronic computing device may comprise electronic means, for example processors, circuits, in particular integrated circuits, and further electronic means, for performing a method according to the preceding aspect.

[0016] Furthermore, the invention relates to a motor vehicle comprising at least the assistance system.

[0017] Advantageous forms of configuration of the method are to be regarded as advantageous forms of the assistance system as well as of the motor vehicle. The assistance system as well as the motor vehicle therefore comprises means for performing the method.

[0018] Further advantages, features, and details of the invention derive from the following description of a preferred embodiment as well as from the drawing. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figure and/or shown in the figure alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWING

[0019] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawing, which is incorporated in and constitutes a part of this disclosure, illustrates exemplary embodiments and together with the description, serves to explain the disclosed principles. The same numbers are used throughout the figure to reference like features and components. Some embodiments of system and/or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figure.

[0020] The drawing shows in: [0021] Fig. 1 a schematic top view of an embodiment of a motor vehicle comprising an embodiment of an assistance system.

[0022] In the figure the same elements or elements having the same function are indicated by the same reference signs.

DETAILED DESCRIPTION

[0023] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0024] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0025] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by "comprises" or "comprise" does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0026] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0027] Fig. 1 shows a schematic top view of an embodiment of a motor vehicle 10 comprising at least an assistance system 12. The motor vehicle 10 is shown in a first time step ti and in a second time step t2. According to the shown embodiment, the motor vehicle 10 has driven a distance I between the first time step ti and the second time step t2.

[0028] The assistance system 12 is configured for generating a three-dimensional model 14 of the surroundings 16 of the motor vehicle 10. According to the shown embodiment, the surroundings 16 may comprise a tree 18.

[0029] Furthermore, the assistance system 12 comprises at least one capturing device 20 and one electronic computing device 22. The electronic computing device 22 may comprise at least processors, circuits, and further electronic means for performing steps of the method. Furthermore, the assistance system 12 may comprise a display device 24.

[0030] According to Fig. 1, an external electronic computing device 26 is shown.

[0031] According to an embodiment of the method for generating the three-dimensional model 14 at least a first image 28 of the surroundings 16 is captured in a first time step ti and a second image 30 of the surroundings 16 in the second time step t2 is captured by the capturing device 20, wherein the capturing device 20 may be for example a front camera of the motor vehicle 10. The second time step t2 is after the first time step ti, and the second image 30 shows the surroundings 16, for example the tree 18, from a different perspective than the first image 28. At least the first image 28 and the second image 30 are rendered to the three-dimensional model 14 by using a neural radiance field method by the electronic computing device 22 of the assistance system 12.

[0032] In particular, a reasoned advantage in the three-dimensional scene reconstruction allows to generate photorealistic renderings of the scene based on a small set of images from different perspectives, for example the neural radiance fields (NeRF). Therefore, the assistance system 12 is described in the motor vehicle 10 with for example the front-facing camera. The assistance system 12 captures for example images 28, 30 every five meters of a driven distance to generate a set of images 28, 30 from different perspectives. The assistance system 12 uses a three-dimensional scene reconstruction technique, such as the neural radiance field method, to generate the three-dimensional model 14 of the driven route. This three-dimensional model 14 is used to render a visualization of the surroundings 16 based on the current position and orientation of the motor vehicle 10. This visualization is displayed to the driver, for example on a display device 24, to give additional guidance in bad weather conditions or to serve as a basis for further environmental annotations. Also, the three-dimensional model 14 may be conditioned to visualize the scene in different lighting, seasons or weather conditions, which could serve as a theme or gimmick for the driver. The generated three-dimensional model 14 of the route may be uploaded to a backend, which is shown with the external electronic computing device 26, and may be aggregated over all motor vehicles in the fleet to generate a further three-dimensional model 32 of all driven roads in the world. When a driver intends to drive a new route, the three-dimensional model 14, 32 may be downloaded from the external electronic computing device 26 to the motor vehicle 10 or to a smartphone. The driver may virtually drive the route before actually driving in the motor vehicle 10 to become familiarized with the corresponding environment.

Reference Signs motor vehicle 12 assistance system 14 three-dimensional model 16 surroundings 18 tree capturing device 22 electronic computing device 24 display device 26 external electronic computing device 28 first image second image 32 further three-dimensional model ti first time step t2 second time step

Claims

SCLAIMS1. A method for generating a three-dimensional model (14) of the surroundings (16) of a motor vehicle (10) by an assistance system (12) of the motor vehicle (10), comprising the steps of: - capturing at least a first image (28) of the surroundings (16) in a first time step (ti) and a second image (30) of the surroundings (16) in a second time step (t2) by a capturing device (20) of the assistance system (12), wherein the second time step (t2) is after the first time step (t1), and wherein the second image (30) shows the surroundings (16) from a different perspective than the first image (28); and - rendering at least the first image (28) and the second image (30) to the three-dimensional model (14) by using a neural radiance field method by an electronic computing device (22) of the assistance system (12).
2. The method according to claim 1, characterized in that a position and/or an orientation of the motor vehicle (10) during capturing the images (28, 30) is taken into consideration for generating the three-dimensional model (14).
3. The method according to claim 1 or 2, characterized in that the three-dimensional model (14) is transmitted to an external electronic computing device (26).
4. The method according claim 3, characterized in that the three-dimensional model (14) is used for generating a further three-dimensional model (32) of further surroundings by the external electronic computing device (26).
5. The method according claim 3 or 4, characterized in that the three-dimensional model (14) and/or the further three-dimensional model (32) is provided for further motor vehicles.
6. The method according to any one of claims 1 to 5, characterized in that the three-dimensional model (14) is displayed on a display device (24) of the assistance system (12).
7. A computer program product with program code means for performing a method according to any one of claims 1 to 6.
8. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 7.
9. An assistance system (12) for generating a three-dimensional model (14) of the surroundings (16) of a motor vehicle (10), comprising at least one capturing device (20) and one electronic computing device (22), wherein the assistance system (12) is configured for performing a method according to any one of claims 1 to 6.