DE102020209910A1 - Method for generating a rendered three-dimensional model of a vehicle - Google Patents

Abstract

A method for generating a rendered three-dimensional model of a vehicle is proposed, comprising the following steps:
providing visual images that depict surroundings of the vehicle with a viewing angle directed laterally from the vehicle;
providing at least one visual sky image that depicts the surroundings of the vehicle with a viewing angle pointing upwards from the vehicle;
generating a full sphere image of the surroundings of the vehicle for rendering the model, using the provided visual images and the at least one visual sky image, wherein the full sphere image spatially includes the model for rendering;
generating at least one reflection of the environment of the vehicle on a surface of the model of the vehicle using the full-sphere image to render a rendered three-dimensional model of the vehicle.

Description

State of the art

The automation of driving goes hand in hand with equipping vehicles with increasingly extensive and powerful sensor systems for detecting the surroundings.

Current state-of-the-art surround viewing capabilities, based in particular on video images of the surroundings, render a view of a 3D scene mapped with real-time video data.

The aim of the current 3D surround vision systems is to support the driver during maneuvers so that he can react quickly in the event of danger. The environment is typically imaged using a fisheye camera on each side of the vehicle in order to generate the surround view. A 3D model of a vehicle can be placed within such a generated surround view in order to create a bird's eye view of the respective scene from any point of view using a combination of the 3D model of the vehicle and the surround view. Such a bird's-eye view can support a driver of the vehicle, for example during parking maneuvers, in particular when the vehicle is brought close to an object.

Disclosure of Invention

Typically, the rendered 3D model of the vehicle is not photo-realistic, for example there are no reflections from the surroundings or refractions of light on the surfaces of the 3D model. Typically, gloss effects and frozen reflections (baked refection map) are simulated on the surface of the 3D model using virtual light sources. This does not create a realistic impression of the 3D model of the vehicle.

According to aspects of the invention, a method for generating a rendered three-dimensional model of a vehicle, a rendering device, a vehicle, a use of the method, a computer program and a machine-readable storage medium are proposed according to the features of the independent claims. Advantageous configurations are the subject of the dependent claims and the following description.

Throughout this description of the invention, the sequence of method steps is presented in such a way that the method is easy to follow. However, those skilled in the art will recognize that many of the method steps can also be carried out in a different order and lead to the same or a corresponding result. In this sense, the order of the method steps can be changed accordingly. Some features are numbered to improve readability or to clarify attribution, but this does not imply the presence of specific features.

• In einem Schritt werden visuelle Bilder bereitgestellt, die mit einem von dem Fahrzeug aus seitlich gerichteten Blickwinkel ein Umfeld des Fahrzeugs abbilden. In einem weiteren Schritt wird zumindest ein visuelles Sky-Bild, das mit einem von dem Fahrzeug aus nach oben gerichteten Blickwinkel das Umfeld des Fahrzeugs abbildet, bereitgestellt. In einem weiteren Schritt wird ein Vollsphären-Bild des Umfeldes des Fahrzeugs zum Rendern des Modells, mittels der bereitgestellten visuellen Bilder und dem zumindest einen visuellen Sky-Bild, generiert, wobei das Vollsphären-Bild das Modell zum Rendern räumlich umfasst. In einem weiteren Schritt wird zumindest eine Reflexion des Umfeldes des Fahrzeugs auf einer Oberfläche des Modells des Fahrzeugs mittels des Vollsphären-Bildes generiert, um ein gerendertes dreidimensionales Modell des Fahrzeugs zu rendern.

According to one aspect, a method for generating a rendered three-dimensional model of a vehicle is proposed, which has the following steps:

• In one step, visual images are provided, which depict surroundings of the vehicle with a viewing angle directed laterally from the vehicle. In a further step, at least one visual sky image is provided, which depicts the surroundings of the vehicle with a viewing angle pointing upwards from the vehicle. In a further step, a full sphere image of the surroundings of the vehicle for rendering the model is generated using the visual images provided and the at least one visual sky image, the full sphere image spatially encompassing the model for rendering. In a further step, at least one reflection of the surroundings of the vehicle is generated on a surface of the model of the vehicle using the full-sphere image in order to render a rendered three-dimensional model of the vehicle.

• • die Ermittlung der vom virtuellen Betrachter oder von einer virtuellen Kamera aus sichtbaren Objekte (Verdeckungsberechnung)
• • die Simulation des Aussehens von Oberflächen, beeinflusst durch deren Materialeigenschaften (Shading)
• • die Berechnung der Lichtverteilung innerhalb einer Szene, die sich unter anderem durch die indirekte Beleuchtung zwischen Körpern äußert.

Rendering in computer graphics refers to the generation or image synthesis of an image from raw data. Raw data can be geometric descriptions in 2D or 3D space, such as HTML, SVG etc.
When rendering, the following tasks usually have to be solved:

• • the determination of the objects visible from the virtual viewer or from a virtual camera (occlusion calculation)
• • the simulation of the appearance of surfaces, influenced by their material properties (shading)
• • the calculation of the light distribution within a scene, which is expressed, among other things, by the indirect lighting between bodies.

The feature of providing visual images is to be understood broadly and includes the provision of optical camera images, video recordings or other optical representations that characterize the structures of the environment using other imaging methods, such as recordings with a RADAR system and/or a LIDAR and/or a ultrasound system.

The term full-sphere image is to be interpreted broadly and includes images that include a hemisphere located below the three-dimensional model to the extent that the hemisphere located below the three-dimensional model is generated either by visual images that depict the area around the vehicle from a lateral viewing angle, as well as a hemisphere arranged above the three-dimensional model, which is generated in particular with the visual sky image.
Alternatively or additionally, the hemisphere arranged below the three-dimensional model can be generated in that objects in the lower hemisphere of the surroundings of the vehicle are imaged and stored during operation of the vehicle.
The term full-sphere image thus encompasses in particular an image that completely encompasses the model for rendering in order to depict the entire environment of the vehicle.
In particular, the term full-sphere image also includes an image of the surroundings of the vehicle for rendering, which captures a hemisphere located above the vehicle in addition to the part of the lower hemisphere represented by visual images taken from a side view angle of the vehicle are included.

Here, the term full sphere image includes an image that maps the corresponding solid angle around the vehicle for rendering and can have any spatial shape. For example, in addition to a spherical shape, the full sphere image may have a shape of a box enclosing the vehicle for rendering.

Throughout the description, an image of/of the area surrounding the vehicle with a lateral and/or upward viewing angle is to be interpreted in such a way that a corresponding part of the area is imaged depending on a particular viewing angle and a particular orientation of the viewing angle.

Advantageously, with the generated full-sphere image of the surroundings of the vehicle, the rendered three-dimensional model of the vehicle can appear more photorealistic, in particular due to reflections on the surface of the model. In particular, the three-dimensional model with the reflections generated in this way can implicitly visualize a time of day, ie in particular daylight or nighttime lighting, and the reflections can also characterize the position of the sun or weather-related lighting conditions.
Advantageously, this eliminates the indirect determination of the incidence of light or of a day or night mode used in the prior art, which must be assigned by means of the time of day.

Because a complete full sphere is generated that encloses the vehicle, the entire environment, i. H. including underground and buildings, which are displayed as reflections on the surface of the three-dimensional model of the vehicle, depending on the size of the viewing angle of the camera systems used. Depending on the updating of the visual images, these reflections and also optical refractions can be determined in real time for the surface of the three-dimensional model and rendered accordingly.

This method differs from a skybox, which simulates a horizon in computer graphics, in which a full-sphere image of the vehicle's surroundings is generated in this method in order to use the correspondingly generated full-sphere image and a mapping rule to reflect reflections on the surface of the three-dimensional model of the vehicle.

Using the full-sphere image, a respective reflex can be easily implemented and calculated according to the normal vector on the surface of the three-dimensional model and the respective viewing direction of a virtual camera position.
When using camera systems that can process a high level of dynamics in light intensity (e.g. HDR cameras), the natural dynamics of the light situation can be detected using the full-sphere image and accordingly as reflections or optical refraction effects on the Surface of the three-dimensional model of the vehicle are rendered.

Lateral reflections of the surroundings can only be mapped undistorted onto the surface of the three-dimensional model of the vehicle if the vehicle is arranged within the completely generated full-sphere image, since only laterally aligned camera systems cannot generate an undistorted full-sphere image in real time can be.

According to one aspect, it is proposed that the at least one reflection is generated by mapping the full-sphere image onto the surface of the rendered model.
Such a mapping can, for example, be based on the laws of geometrical optics and, depending on a respective viewing direction of a virtual camera position, assign a reflection of the full-sphere image to this surface point of the three-dimensional model according to a respective normal vector on each point of the surface.

According to one aspect, it is proposed that the at least one sky image is generated using a sky camera system of a vehicle whose model is being rendered, and the sky camera system is set up to have at least one visual sky image to depict the surroundings of the vehicle from an upward viewing angle from the vehicle.
A visual sky image can be generated particularly easily with such a sky camera system.

According to one aspect, it is proposed that the sky camera system is set up to generate a sky image that depicts the entire environment above the vehicle.

A sky image generated in this way characterizes an environment above the vehicle that covers an angular range of 180° and has its origin at the highest point of the vehicle. This can be generated with a fisheye camera system, for example.
Using such a sky camera system allows to generate the full sphere image with little distortion.

According to one aspect, it is proposed that the at least one sky image is generated using at least one image area of images from a camera system, the camera system being set up to image the vehicle's surroundings with a viewing angle directed laterally from the vehicle. Advantageously, no additional camera system needs to be installed in the vehicle and the generation of the full-sphere image can be generated, for example, by sequentially generated images with a different alignment of the camera system due to a movement of the vehicle.

According to one aspect, it is proposed that the at least one image area depicts a viewing angle directed upwards from the vehicle.

According to one aspect, it is proposed that the visual images, which depict the surroundings of the vehicle with a viewing angle directed laterally from the vehicle, are generated by means of four vehicle camera systems, each of which covers at least one angular range of the four sides next to the vehicle depict.
Camera systems that are already provided for a surround view system in the vehicle can advantageously be used for this purpose.

According to one aspect, a rendering device is proposed that is set up to carry out one of the methods described above. With such a device, the corresponding method can easily be integrated into different systems.

According to one aspect, it is proposed that the rendering device has four vehicle camera systems, each of which is set up on a different side of a vehicle to generate images of an area surrounding the vehicle with a viewing angle directed laterally from the vehicle; and provide the images. In addition, the rendering device has at least one sky camera system that is set up to generate at least one visual sky image of the surroundings of the vehicle with an upward viewing angle from the vehicle; and provide the at least one visual sky image.

A vehicle is proposed which has a sky camera system, the sky camera system being set up to produce a sky image, which depicts the entire environment above the vehicle, with a viewing angle pointing upwards from the vehicle Rendering a model of the vehicle with reflections of the environment.

A use of one of the methods described above for supporting a driver of the vehicle of the rendered model when driving the vehicle is proposed.

According to one aspect, a computer program is provided which comprises instructions which, when the computer program is executed by a computer, cause the latter to execute one of the methods described above. Such a computer program enables the method described to be used in different systems.

A machine-readable storage medium is specified, on which the computer program described above is stored. The computer program described above can be transported by means of such a machine-readable storage medium.

exemplary embodiments

• 1 ein Fahrzeug, das mit einer Mehrzahl von Kamerasystemen ausgerüstet ist, um unterschiedliche Winkelbereiche der Umgebung des Fahrzeugs abzubilden; und
• 2 ein dreidimensionales Modell eines Fahrzeugs in einer Umgebung mit Reflexen an der Oberfläche.

Embodiments of the invention are described with reference to FIG 1 until 2 shown and explained in more detail below. Show it:

• 1 a vehicle that is equipped with a plurality of camera systems in order to image different angular ranges of the surroundings of the vehicle; and
• 2 a three-dimensional model of a vehicle in an environment with reflections on the surface.

the 1 outlines a vehicle 100, which is equipped with a plurality of camera systems 110, 120, 130, 140, 150 in order to depict different angular ranges of the surroundings of the vehicle. The camera systems 110, 120, 130, 140, 150 each form a viewing angle 110a, 120a, 130a, 140a, 150a around the surroundings of the vehicle. The viewing angles of the laterally arranged camera systems 110, 120, 130, 140 are directed laterally from the vehicle in order to image an area surrounding the vehicle that is essentially laterally adjacent. The camera system 150 is set up to generate a visual sky image that depicts the surroundings of the vehicle with a viewing angle pointing upwards from the vehicle.

the 2 sketches a three-dimensional model 200 of a vehicle 100 in an environment with reflections 210, 220, 230, 240 on the surface. Objects 210a (skyscraper), 230a (tree), 240a (road) in the surroundings of vehicle 100 are formed as respective reflections on the surface of three-dimensional rendered model 200 of vehicle 100 . The reflex 220 reflects a light source, such as the sun.

Claims (13)

Method for generating a rendered three-dimensional model (200) of a vehicle (100), with the steps: providing visual images that depict an area surrounding the vehicle with a viewing angle (110a, 120a, 130a, 140a) directed laterally from the vehicle (100); providing at least one visual sky image that depicts the surroundings of the vehicle (100) with an upward viewing angle (150a) from the vehicle (100); Generating a full sphere image of the surroundings of the vehicle (100) for rendering the model (200) using the provided visual images and the at least one visual sky image, wherein the full sphere image spatially includes the model (200) for rendering; Generating at least one reflection (210, 220, 230, 240) of the surroundings of the vehicle (100) on a surface of the model (200) of the vehicle (100) using the full-sphere image to create a rendered three-dimensional model (200) of the vehicle (100) to render. procedure according to claim 1 , wherein the at least one reflection (210, 220, 230,240) is generated by mapping the full-sphere image onto the surface of the rendered model (200). procedure according to claim 1 or 2 , wherein the at least one sky image is generated by means of a sky camera system (150) of the vehicle (100), whose model is rendered, and the sky camera system (150) is set up, the at least one visual sky - Image depicting the surroundings of the vehicle with a viewing angle (150a) pointing upwards from the vehicle (100). procedure according to claim 3 , wherein the sky camera system (150) is set up to generate a sky image that depicts the entire environment above the vehicle (100). procedure according to claim 1 or 2 , wherein the at least one sky image is generated by means of at least one image area of images from a camera system (110, 120, 130, 140), the camera system (110, 120, 130, 140) being set up with one of to image the surroundings of the vehicle (100) from the vehicle (100) from a laterally directed viewing angle (110a, 120a, 130a, 140a). procedure according to claim 5 , wherein the at least one image area depicts an upward viewing angle from the vehicle (100). Method according to one of the preceding claims, wherein the visual images, which depict an environment of the vehicle (100) with a vehicle (100) from a laterally directed viewing angle (110a, 120a, 130a, 140a), using four vehicle camera systems (110, 120, 130, 140) are generated, each depicting at least one angular area of the four sides next to the vehicle. Rendering device that is set up according to one of the methods claim 1 until 7 to execute. Rendering device according to claim 8 , comprising: four vehicle camera systems (110, 120, 130, 140), each installed on a different side of a vehicle, input images with a viewing angle (110a, 120a, 130a, 140a) directed laterally from the vehicle generate environment of the vehicle (100); and provide the images; At least one sky camera system (150), which is set up to generate at least one visual sky image of the area surrounding the vehicle (100) with an upward viewing angle (150a) from the vehicle (100); and provide the at least one visual sky image. Vehicle (100) having a sky camera system (150), wherein the sky camera system (150) is set up to produce a sky image with a viewing angle (150a) directed upwards from the vehicle (100). , which maps the entire environment above the vehicle (100) to render a model (200) of the vehicle with reflections of the environment. Use of the method according to any one of Claims 1 until 7 , for assisting a driver of the vehicle of the rendered model in driving the vehicle (100). Computer program, comprising instructions which, when the computer program is executed by a computer, cause the latter to carry out the method according to one of Claims 1 until 7 to execute. Machine-readable storage medium on which the computer program claim 12 is saved.

Images