DE102021202454A1 - Environment modeling based on camera data - Google Patents

Abstract

The present invention relates to a device (18) for generating a 3D model of an environment (12), having: an input interface (26) for receiving image data from an image sensor (16), the image data having a plurality of images of the same area of the environment different focus include; an evaluation unit (28) for detecting a focused image area in an image (36) of the plurality of images; a distance unit (30) for determining depth information for the focused image area based on a previously known assignment of distances to focussing; and a modeling unit (32) for determining three-dimensional position information on objects (22) in the environment based on focused image areas in the images of the plurality of images and determined depth information. The present invention further relates to a system (10) and a method for generating a 3D model of an environment (12), as well as a vehicle and a computer program product with program code for carrying out the steps of the method.

Description

The present invention relates to a device for generating a 3D model of an environment. The present invention further relates to a method and a system for generating a 3D model of an environment. The invention also relates to a vehicle and a computer program product.

Modern vehicles (cars, trucks, motorcycles, etc.) have a variety of sensors (radar, lidar, camera, ultrasound, etc.) that provide information to a vehicle operator or a vehicle's control system. The environment of the vehicle and objects in this environment (other vehicles, infrastructure objects, people, moving objects, etc.) are detected via such environment sensors. Based on the recorded data, a model of the vehicle environment can be generated and changes in this vehicle environment can be reacted to. In particular, a driving function of the vehicle can be carried out partially or fully autonomously.

The use of cameras for environment recognition is becoming more and more important. Due to technical developments, high-resolution camera systems can now be installed at reasonable costs and in partially or fully autonomous vehicles. Advanced image processing and evaluation algorithms can be used to obtain information about objects in the vehicle's surroundings.

A disadvantage of using cameras is the lack of availability of depth information due to the measurement principle. In order to be able to determine distances to objects, image analysis algorithms and, in particular, machine learning approaches can be used. As an alternative or in addition, so-called stereo camera systems can be used, in which at least two image sensors, which are spatially separated from one another, record images and these images are then evaluated together. This entails higher costs due to the need to use additional hardware. These additional costs are particularly relevant when it is taken into account that in most cases several areas in the vicinity of a vehicle are to be detected by sensors, which require cameras aligned in different directions.

In this context, in DE 10 2018 221 133 A1 disclosed a position and motion detection using an autofocus system. In particular, a device is disclosed which is designed to estimate position information, movement information and/or shape information relating to an object by means of autofocusing of image data. In particular, it is proposed to carry out a distance measurement with respect to an object by means of focusing, with an image distance being varied until a maximum of image sharpness is reached. A distance relative to an object is then determined on the basis of this image distance.

Disadvantages of previous approaches are, in particular, the high demand for computing power and the long calculation time. Especially in situations in which a specific object is not to be viewed specifically, but instead a detection of various objects in an environment is to be carried out first, such approaches cannot be used in some cases.

Proceeding from this, the object of the present invention is to provide an approach for generating a 3D model of an environment based on data from a camera. In particular, it should be possible to generate a 3D model based on image data from a single image sensor. The 3D model generated should contain information on the positions and distances of objects in the field of view of the image sensor.

• einer Eingangsschnittstelle zum Empfangen von Bilddaten eines Bildsensors, wobei die Bilddaten eine Mehrzahl von Bildern desselben Bereichs der Umgebung mit unterschiedlicher Fokussierung umfassen;
• einer Auswerteeinheit zum Detektieren eines fokussierten Bildbereichs in einem Bild der Mehrzahl von Bildern;
• einer Abstandseinheit zum Ermitteln einer Tiefeninformation für den fokussierten Bildbereich basierend auf einer vorbekannten Zuordnung von Abständen zu Fokussierungen; und
• einer Modellierungseinheit zum Ermitteln von dreidimensionalen Positionsangaben zu Objekten in der Umgebung basierend auf fokussierten Bildbereichen in den Bildern der Mehrzahl von Bildern und ermittelten Tiefeninformationen.

To solve this problem, the present invention relates in a first aspect to a device for generating a 3D model of an environment, with:

• an input interface for receiving image data from an image sensor, the image data comprising a plurality of images of the same area of the environment with different focus;
• an evaluation unit for detecting a focused image area in an image of the plurality of images;
• a distance unit for determining depth information for the focused image area based on a previously known assignment of distances to focal points; and
• a modeling unit for determining three-dimensional position information on objects in the environment based on focused image areas in the images of the plurality of images and determined depth information.

• einer Kamera mit einem Bildsensor zum Erfassen der Umgebung, wobei die Kamera eine senkrecht zur Bildebene gegenüber dem Bildsensor bewegliche Linse zum Verändern einer Fokussierung umfasst und wobei der Bildsensor zum Aufnehmen einer Mehrzahl von Bildern an unterschiedlichen Positionen der Linse ausgebildet ist; und
• einer Vorrichtung wie zuvor beschrieben.

In a further aspect, the present invention relates to a system for generating a 3D model of an environment, comprising:

• a camera with an image sensor for capturing the environment, wherein the camera comprises a lens that can be moved perpendicularly to the image plane relative to the image sensor for changing a focus, and wherein the image sensor configured to capture a plurality of images at different positions of the lens; and
• a device as previously described.

Further aspects of the invention relate to a vehicle with a system as described above, a method designed according to the device and a computer program product with program code for carrying out the steps of the method when the program code is executed on a computer. In addition, one aspect of the invention relates to a storage medium on which a computer program is stored which, when executed on a computer, causes the method described herein to be carried out.

Preferred developments of the invention are described in the dependent claims. It goes without saying that the features mentioned above and those still to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention.

In particular, the device, the system, the vehicle, the method and the computer program product can be implemented in accordance with the configurations defined for the device and the system in the dependent claims.

According to the invention, it is provided that data from an image sensor is received. This image data includes several images that were recorded with different focuses. For example, it is possible to generate such a plurality of images by moving the lens and the image sensor perpendicularly to the image plane, ie by changing the focus. In this case, the distance is preferably changed quickly, for example by means of an oscillating lens, in order to be able to record the same image detail several times. In each image of the plurality of images, a focused image area, that is to say an image area that is in sharp focus, is then determined. This focused image area is evaluated in order to determine depth information based on a previously known assignment of distances or depths to focussing of the camera. The previously known association corresponds to an association, known for a camera, between a distance between the lens and the image sensor and a distance of a focused or in-focus area from the camera for this distance. The assignment can be determined, for example, in a calibration or can also be determined theoretically in a corresponding calculation. According to the invention, a 3D model of the environment is then determined in a modeling unit. The model includes position information for an object, which contains two-dimensional information about a position within the image and depth information, ie information about the distance from the image sensor. In particular, a cloud of points, ie information about a position of a whole series of points in a three-dimensional reference coordinate system, can be provided.

In comparison to previous approaches of camera-based generation of a 3D model, only a single camera or a single image sensor is required according to the invention. It is not necessary to use multiple image sensors in the sense of a stereo camera arrangement. Installation space and costs can be saved. In comparison to approaches of generating a 3D model through image recognition and corresponding algorithmic evaluation approaches, a distance to objects, ie a depth of a point, can be determined with greater reliability. In comparison to using distance-based sensors, such as radar, lidar or ultrasonic sensors, depth information can also be obtained using a purely camera-based system. Costs can be saved.

In a preferred embodiment, the input interface is designed to receive the image data from a camera with a lens that can be moved perpendicularly to an image plane of the image sensor relative to the image sensor in order to change a focus. In particular, it is possible for the camera to be focused by changing a distance between the image sensor and the lens. Such a change in distance can be achieved, for example, by modifying a conventional autofocus system.

In a preferred embodiment, the evaluation unit is designed to detect the focused image area based on a contrast evaluation and/or based on a detection of a phase of the image. In particular, it is possible to determine which area of the image is sharp, ie in focus, by examining a contrast profile within the image. In particular, focused edges or similar features can be recognized. A phase can also be evaluated in order to identify focused image areas based on this. Efficient data evaluation can be enabled by the automatic identification of focused image areas. Fast processing is achieved.

In a preferred embodiment, the distance unit for determining the depth information for the focused image area is based on a corresponding image area is formed in another image of the plurality of images. When determining the depth information, another image from the plurality of images, ie an image with a different focus, is also taken into account in addition to the current image. Additional information can be obtained by comparing two images or the focused areas of both images. As a result, the accuracy when determining the depth information can be improved.

In a preferred embodiment, the distance unit is designed to determine the depth information based on a change in contrast of the focused image area and a corresponding image area in another image of the plurality of images. In particular, a corresponding image area, ie the same image area as the focused image area in the currently viewed image, can also be evaluated in another image. Depending on the difference in the contrast values of both image areas, the accuracy when determining the depth information can be further improved. A high level of accuracy and reliability is achieved when determining the depth information.

In a preferred embodiment, the distance unit is designed to determine the depth information for the focused image area based on a non-focused image area in the same image of the plurality of images. The unfocused image area is preferably directly adjacent to the focused image area. Additionally or alternatively, a further improvement in the accuracy when determining the depth information can be achieved by additionally considering other areas within the currently evaluated image of the plurality of images. In addition to the focused area, a further image area is therefore considered when determining the depth information. Depending on the type of image and the currently evaluated objects, for example, blurred, i.e. not focused, objects that are nevertheless recognizable can be taken into account. The accuracy in determining the depth information is further improved.

In a preferred embodiment, the distance unit is designed to determine the depth information based on a pre-trained artificial neural network. In addition to using the previously known mapping, a further improvement in accuracy can be achieved by using a pre-trained artificial neural network. Corresponding training data contain an assignment of image information to depths or distances. Such training data can be generated, for example, by manual annotation, by simulatively generated data or by a combination with another sensor system (for example lidar or radar sensors). The accuracy in determining the depth information can be further improved by the additional use of a pre-trained artificial neural network.

In a preferred refinement, the input interface is designed to receive movement data with information on a relative movement between the environment and the image sensor during the recording of the plurality of images. The modeling unit is designed to compensate for a movement of the image sensor in relation to the environment based on the movement data. In order to compensate for a possible rapid movement of the image sensor in relation to the surroundings, for example when the image sensor is attached to a vehicle, movement data can also be taken into account in the image evaluation. Movement data can be measurement data from an inertial sensor, for example. It is also possible for the movement to be determined using the images themselves or based on sensor data from other sensors (e.g. radar or lidar). The movement data can be used to correct the movement. In particular, back calculation of the image data to a common reference point can be made possible. The determination of the three-dimensional position information can be improved by compensating for the movement. In particular, the accuracy can be increased.

In a preferred embodiment, the input interface is designed to receive the previously known assignment. Additionally or alternatively, the previously known association includes a look-up table, which was preferably determined in a calibration of a camera. Specifically, a look-up table maps lens-to-imager distances to imager-to-in-focus object distances. It is possible that the previously known assignment is received via the input interface. In this respect, the previously known assignment can be updated. In this case, in particular, a camera calibration can be used to generate the previously known association.

In a preferred embodiment of the system, the camera includes a microelectromechanical actuator for moving the lens relative to the image sensor. Efficient manufacturability can be achieved by using a MEMS actuator while enabling precise motion control.

An object in the surroundings of a vehicle can in particular be another vehicle, a cyclist, a pedestrian, an animal or a static object such as a car tire lying on the road, etc. An environment can in particular be an area surrounding a vehicle or an area that can be seen from the vehicle. Image data from an image sensor include, in particular, color values of pixels. Focusing means focusing. A focused area is an image area that is in focus. Depth information is information about a distance of an object within the image from the image sensor. A three-dimensional position statement can be in different coordinate systems. In particular, a camera-fixed coordinate system can be used and an indication of a position of the pixel within the image and a distance of the pixel from the image sensor can be understood as a three-dimensional position indication. A camera or camera sensor comprises at least one image sensor and one lens.

• 1 eine schematische Darstellung eines erfindungsgemäßen Systems zum Erzeugen eines 3D-Modells einer Umgebung;
• 2 eine schematische Darstellung einer erfindungsgemäßen Vorrichtung zum Erzeugen eines 3D-Modells einer Umgebung;
• 3 eine schematische Darstellung des Veränderns einer Fokussierung durch Ändern des Abstands zwischen Linse und Bildsensor;
• 4 eine schematische Darstellung eines Ansatzes zum Detektieren eines fokussierten Bildbereichs;
• 5 eine schematische Darstellung eines weiteren Ansatzes zum Detektieren eines fokussierten Bildbereichs; und
• 6 eine schematische Darstellung eines erfindungsgemäßen Verfahrens.

The invention is described and explained in more detail below using a few selected exemplary embodiments in connection with the accompanying drawings. Show it:

• 1 a schematic representation of an inventive system for generating a 3D model of an environment;
• 2 a schematic representation of a device according to the invention for generating a 3D model of an environment;
• 3 a schematic representation of changing a focus by changing the distance between the lens and the image sensor;
• 4 a schematic representation of an approach for detecting a focused image area;
• 5 a schematic representation of a further approach for detecting a focused image area; and
• 6 a schematic representation of a method according to the invention.

In the 1 a system 10 according to the invention for generating a 3D model of an environment 12 is shown schematically. The system includes a camera 14 with an image sensor 16 and a device 18 for generating a 3D model of the environment 12. In the exemplary embodiment shown, the system 10 is integrated into a vehicle 20. The representation is to be understood as a lateral sectional view of the vehicle 20 on a roadway. In the illustrated exemplary embodiment, the environment 12 includes in particular an area in front of the vehicle 20 in which an object 22 is located. The object 22 can, for example, be another vehicle or also a vehicle part lying on the road. The device 18 can be integrated, for example, in a control unit or in a central computer of the vehicle 20 or also in the camera 14 . In particular, the camera 14 can be mounted on the vehicle 20 . In the representation in the 1 the field of view 24 of the camera is indicated by dashed lines as an example.

According to the invention, it is provided that the device 18 is used to evaluate image data from the image sensor 16 . In this case, the image data comprise a plurality of images of the same area of the surroundings 12 which were recorded with different focusing. This different focusing can be achieved, for example, by varying a distance between the image sensor 16 and a lens. According to the invention, distances to objects in the environment are inferred based on different focused regions to be recognized in sharp focus in the images of the plurality of images. Based on this, a 3D model of the environment 12 is generated.

In the 2 a device 18 according to the invention for generating a 3D model of an environment is shown schematically. The device 18 includes an input interface 26, an evaluation unit 28, a distance unit 30 and a modeling unit 32. The units and interfaces can be partially or completely implemented in software and/or in hardware. In particular, the units can be designed as processors, processor modules or also as software for a processor. The device 18 can be designed in particular in the form of a control unit of an autonomous or semi-autonomous vehicle or as software for a control unit of an autonomous or semi-autonomous vehicle.

Data from the image sensor are received via the input interface 26 . In this case, the image sensor is in particular an image sensor of a camera which is preferably arranged on a vehicle. The camera takes pictures of an area of the environment and transmits them to the device 18. The data can be transmitted, for example, via a vehicle bus system to which the input interface 26 is connected. The image data includes a plurality of images. These different images each show the same area but are taken with different focus. In particular, the images are recorded at different distances between the lens and the image sensor.

In the evaluation unit 28, an image region that is focused in this image is detected in one image of the plurality of images. Different algorithmic approaches can be used for this will. For example, contours or contrasts can be evaluated. It is also possible to analyze a phase of the image (phase evaluation) in order to determine which areas of the image are sharp, i.e. focused. In particular, points can be detected as focused image areas.

In the distance unit 30, depth information is then determined for this focused image area. In this context, depth information is understood to mean a distance of the focused image area from the image sensor or from the camera. A previously known assignment of distances between objects or sharply set, focused image areas and focusing of the camera is used to determine this depth information. A focusing of the camera is understood to mean a setting of the camera focus system. In particular, focusing of the camera is understood to mean a distance between the lens and the image sensor. It is possible to determine such a previously known assignment in a calibration of a camera. During the calibration, objects are arranged at different distances from the image sensor, then images can be recorded with different focusing and it can be detected which focusing leads to sharp objects at which distances. For example, the previously known assignment can be in the form of a look-up table.

It is also possible for other images of the plurality of images to be accessed in the distance unit 30 for determining the depth information. In particular, the depth information can be determined based on a corresponding image area in the other image. A corresponding image area designates a corresponding image section, that is to say a corresponding section or a corresponding point within the plurality of images. In particular, a change in contrast between the focused image area and the corresponding image area in the other image can be viewed and evaluated for the determined depth information.

Furthermore, it is possible that the distance unit 30 is designed to determine the depth information based on the focused area and an additional consideration of a non-focused other image area in the same image. For example, information on the distance between objects in an image that are not in focus or not in focus can be obtained using image evaluation algorithms. This information can then be used as additional information when determining the depth information.

Additionally or alternatively, the distance unit 30 can also be designed to use a pre-trained artificial neural network in order to determine the depth information. Based on previous experiences that were taken into account in a training phase, this pre-trained artificial neural network can enable the depth information to be determined more precisely. For example, features of the focused image area or also features of other images or features from non-focused image areas can be used for this purpose.

A 3D model of the environment is created in the modeling unit 32 on the basis of the focused image areas and the determined depth information.

In particular, three-dimensional position information on objects in the area is determined for this purpose. This position information can be output, for example, in the form of a two-dimensional position within the image in combination with metric distance information that corresponds to the depth information. The three-dimensional position information provided by the modeling unit 32 can then be further used in algorithms for controlling an autonomous vehicle or also in a driver assistance system.

In the 3 Schematically illustrates the concept of changing the focus. A lens 34 is reciprocated in a rapid motion (back and forth relative to the image sensor 16). On the left is in the 3 while the distance between lens 34 and image sensor 16 greater than on the right side. This means that the object 22 in the image 36 on the left side is not in focus, ie it is not in focus, whereas a focused object 22 can be seen within the image 36 on the right side. By changing the distance between lens 34 and image sensor 16, multiple images of an area can be captured. For example, a fast movement of the lens 34 can be produced by using a stepper motor, a piezo actuator or a microelectromechanical system (MEMS).

According to the invention, a distance between the image sensor and a point in the image is calculated based on the sharpness in different points of the generated plurality of images. One of the plurality of images has different focused areas depending on the position of the lens 34 at which that image was taken. In this respect, the approach enables a kind of layered view of the environment. Starting from these layers, in a In the simplest case, a 3D model of the environment can be constructed in the modeling unit, for example, insofar as a distance specification is added for each area or for each point, starting from detected focused areas. A kind of point cloud is generated in this respect.

In the 4 an approach for detecting focused image areas or focused points in an image based on a phase evaluation is shown schematically. From left to right in the 4 the distance between lens 34 and image sensor 16 increases. The displayed image 36 or the image section or point at the top of the building is only shown sharply in the second case. All other cases result in blurred images, as shown in the scene image 38 on the image sensor. In this respect, a statement can be made by detecting a phase as to whether the image area is in focus. The sharpness of the image can also be determined using a so-called PDAF method (Phase Detection Auto Focus).

In the 5 an approach for detecting a focused image area based on a contrast evaluation is shown schematically. The distance between lens 34 and image sensor 16 is in focus only in the upper case. Only in this case is a clear peak reached in the contrast representation 40 shown on the right-hand side. The other cases result in less pronounced contrasts. This makes it possible to detect focused image areas within images.

Motion information may be used to compensate for motion performed by the camera during the short period of time of generating the different images of the plurality of images. Such movement information can be generated, for example, by means of an inertial sensor. However, it is also possible for the movement, ie the relative movement between the image sensor and the environment, to be determined by evaluating different images.

In the 6 an inventive method for generating a 3D model of an environment is shown schematically. The method comprises the steps of receiving S10 image data, detecting S12 a focused image area, determining S14 depth information and determining S16 three-dimensional position information. The method can be implemented in particular in the form of software that runs on a processor of a vehicle or a vehicle control unit. It goes without saying that the method can also be implemented as a smartphone app or as control software for a digital camera.

The invention has been comprehensively described and explained with reference to the drawings and the description. The description and explanation are intended to be exemplary and not limiting. The invention is not limited to the disclosed embodiments. Other embodiments or variations will become apparent to those skilled in the art upon use of the present invention upon a study of the drawings, the disclosure, and the claims that follow.

In the claims, the words "comprising" and "having" do not exclude the presence of other elements or steps. The undefined article "a" or "an" does not exclude the presence of a plural. A single element or unit can perform the functions of several of the units recited in the claims. An element, unit, interface, device, and system may be partially or fully implemented in hardware and/or software. The mere naming of some measures in several different dependent patent claims should not be understood to mean that a combination of these measures cannot also be used to advantage. A computer program may be stored/distributed on a non-volatile medium, such as an optical memory or a solid state drive (SSD). A computer program may be distributed with and/or as part of hardware, for example via the Internet or via wired or wireless communication systems. Reference signs in the claims are not to be understood as restrictive.

Reference List

10

system

12

vicinity

14

camera

16

image sensor

18

contraption

20

vehicle

22

object

24

field of view

26

input interface

28

evaluation unit

30

distance unit

32

modeling unit

34

lens

36

picture

38

scene illustration

40

contrast display

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102018221133 A1 [0005]

Claims (14)

Apparatus (18) for creating a 3D model of an environment (12), comprising: an input interface (26) for receiving image data from an image sensor (16), the image data comprising a plurality of images of the same area of the environment with different focus; an evaluation unit (28) for detecting a focused image area in an image (36) of the plurality of images; a distance unit (30) for determining depth information for the focused image area based on a previously known assignment of distances to focussing; and a modeling unit (32) for determining three-dimensional position information on objects (22) in the area based on focused image areas in the images of the plurality of images and determined depth information. Device (18) after claim 1 , wherein the input interface (26) is designed to receive the image data from a camera (14) with a lens (34) movable perpendicularly to an image plane of the image sensor (16) relative to the image sensor in order to change a focus. Device (18) according to one of the preceding claims, wherein the evaluation unit (28) is designed to detect the focused image area based on a contrast evaluation and/or based on a detection of a phase of the image (36). Device (18) according to one of the preceding claims, wherein the distance unit (30) is designed to determine the depth information for the focused image area based on a corresponding image area in another image of the plurality of images. Device (18) after claim 4 , wherein the distance unit (30) is designed to determine the depth information based on a contrast change of the focused image area and a corresponding image area in another image of the plurality of images. Device (18) according to any one of the preceding claims, wherein the distance unit (30) is designed to determine the depth information for the focused image area based on an unfocused image area in the same image (36) of the plurality of images; and the unfocused image area is preferably directly adjacent to the focused image area. Device (18) according to one of the preceding claims, wherein the distance unit (30) is designed to determine the depth information based on a pre-trained artificial neural network. Device (18) according to any one of the preceding claims, wherein the input interface (26) is designed to receive movement data with information on a relative movement between the environment (12) and the image sensor (16) during the recording of the plurality of images; and the modeling unit (32) is designed to compensate for a movement of the image sensor in relation to the environment based on the movement data. Device (18) according to any one of the preceding claims, wherein the input interface (26) is designed to receive the previously known association; and or the previously known association includes a look-up table, which was preferably determined in a calibration of a camera. A system (10) for creating a 3D model of an environment (12), comprising: a camera (14) with an image sensor (16) for capturing the environment, wherein the camera comprises a lens (34) that can be moved perpendicularly to the image plane relative to the image sensor to change a focus and wherein the image sensor is used to record a plurality of images at different positions of the lens is formed; and a device (18) according to any one of the preceding claims. system (10) according to claim 10 , wherein the camera (14) comprises a microelectromechanical actuator for moving the lens (34) relative to the image sensor (16). Vehicle (20) with a system (10) according to one of Claims 10 and 11 . Method for generating a 3D model of an environment, comprising the steps of: receiving (S10) image data from an image sensor (16), the image data comprising a plurality of images of the same area of the environment with different focusing; detecting (S12) a focused image area in an image (36) of the plurality of images; determining (S14) depth information for the focused image area based on a previously known assignment of distances to focussing; and determining (S16) three-dimensional position information on objects (22) in the area on focused image areas in the images of the plurality of images and determined depth information. Computer program product with program code for performing the steps of the method Claim 13 , if the program code is run on a computer.

Images