DE102019107650A1 - Object shape detection based on two types of environmental sensors - Google Patents

Abstract

The present invention relates to a method for object shape detection based on two types of environment sensors (14, 16) which are used in a driving support system (12) of a vehicle (10), the first type of environment sensor (14, 16) having an image sensor (14) an image field of view (20) and the second type of environment sensor is an area sensor (16) with an area field of view (22) which at least partially overlaps the image field of view (20), with the steps of providing a two-dimensional data point matrix which the environment (18) of the vehicle (10) in the image field of view (20), by at least one image sensor (14), identifying the at least one object (24) in the two-dimensional data point matrix, performing a 2D shape estimation of the identified at least one object (24) based on the two-dimensional data point matrix, Providing a three-dimensional matrix of data points representing the surroundings (18) of the vehicle (10) in the area tfeld (22), by at least one area sensor (16), identifying the at least one object (24) in the three-dimensional data point matrix, executing a 3D shape estimation of the identified at least one object (24) based on the three-dimensional data point matrix and determining the shape of the identified at least one object (24) by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object (24). The present invention also relates to a driving support system (12) for object shape detection based on two types of environment sensors according to the above method.

Description

The present invention relates to a method for object shape detection based on two types of environment sensors which are used in a driving support system of a vehicle, the first environment sensor type being an image sensor with an image field of view and the second environment sensor type being an area sensor with an area field of view which at least partially covers the image field of view overlaps.

The present invention also relates to a driving support system for object shape detection based on two types of environment sensors according to the above method, the driving support system at least one image sensor as the first type of environment sensor for providing a two-dimensional data point matrix which represents the environment of the vehicle in an image field of view, and at least one area sensor as second type of environment sensor for providing a three-dimensional data point matrix which represents the environment of the vehicle in an area field of view, wherein the area field of view at least partially overlaps the image field of view.

Autonomous and semi-autonomous driving is becoming an increasingly important factor in the automotive industry. Prototypes for autonomous driving have already been developed and used and are currently being tested, in some cases even under real driving conditions. Autonomous driving is seen as a breakthrough technology in the automotive sector.

Autonomous and semi-autonomous driving depends on knowing the surroundings of the vehicle, which is also known as the ego vehicle. Various types of environmental sensors can be used in the ego vehicle to monitor its surroundings and to detect objects such as third-party vehicles or traffic signs. For autonomous and semi-autonomous driving systems, the calculation of precise trajectories of moving objects, e.g. Third-party vehicles, an essential requirement in the vicinity of the ego vehicle. Therefore, it is important to determine an exact shape of objects that are detected in a vicinity of the vehicle.

The environmental sensors used to monitor the surroundings of the vehicle can be divided into various categories based on the type of information obtained by the environmental sensors. For example, one type of environment sensor can provide a three-dimensional data point matrix that represents the environment of the vehicle in an area field of view. These environmental sensors, e.g. Environment sensors based on LiDAR or radar provide precise distance information for objects in the vicinity of the ego vehicle and are therefore referred to as area sensors. Another type of environmental sensor, considered to be image sensors, provides a two-dimensional array of data points that represent the surroundings of the vehicle in an image field of view, thereby providing sensor information that is highly suitable for identifying objects.

In order to provide fully autonomous vehicles, different sensor technologies must be used at the same time in order to generate consistent environmental information. Therefore, sensor information from both environmental sensor types is used for autonomous and semi-autonomous driving systems. Sensor information is merged to determine positions of objects in the vicinity of the vehicle. In this connection, it is necessary that the merging is carried out precisely, especially in urban environments. Unreliable information, especially about the distance to an object and about the orientation of the object, reduce the overall performance of the system and increase the risk of accidents.

The same requirements apply to conventional driver assistance systems, e.g. for the implementation of an ACC, Adaptive Cruise Control, function that is typically based on a front radar and / or a front camera as environmental sensors. If both types of environment sensors are used, the sensor information from both environment sensors is merged in order to determine a distance to objects that are located in front of the ego vehicle.

The invention is based on the object of specifying a method for object shape detection and a driving support system for object shape detection of the above-mentioned type, which enable an improved object shape detection.

This problem is solved by the independent claims. Advantageous refinements are given in the subclaims.

In particular, the present invention specifies a method for object shape detection based on two types of environment sensors that are used in a driving support system of a vehicle, the first environment sensor type being an image sensor with an image field of view and the second environment sensor type being an area sensor with an area field of view that is the Image field of view at least partially overlapped, with the steps of: providing a two-dimensional data point matrix which represents the surroundings of the vehicle in the image field of view by at least an image sensor, identifying at least one object in the two-dimensional data point matrix, performing a 2D shape estimation of the identified at least one object based on the two-dimensional data point matrix, providing a three-dimensional data point matrix, which represents the surroundings of the vehicle in the area field of view, by at least one area sensor, identifying at least one Object in the three-dimensional data point matrix and performing a 3D shape estimation of the identified at least one object based on the three-dimensional data point matrix and determining the shape of the identified at least one object by merging the 2D shape estimation with the 3D shape estimation for the identified at least one object.

The present invention also specifies a driving support system for object shape detection based on two types of environment sensors according to the above method, the driving support system at least one image sensor as the first environment sensor type for providing a two-dimensional data point matrix that represents the environment of the vehicle in an image field of view, and at least one Area sensor as a second type of environment sensor for providing a three-dimensional data point matrix which represents the environment of the vehicle in an area field of view, the area field of view at least partially overlapping the image field of view.

The basic idea of the invention is to improve the object detection and processing in a driving support system by providing simple means for carrying out a merging by individually carrying out a 2D / 3D shape estimation of the identified at least one object. This is achieved in that the two-dimensional data point matrix and the three-dimensional data point matrix are processed and the information relating to the two-dimensional data point matrix and the three-dimensional data point matrix, which represents the surroundings of the vehicle in the image field of view, are merged. In this way, reliable shape detection can be carried out. The shape of the at least one identified object can be processed further for various driving support functions.

The vehicle, i.e. the ego vehicle can be any type of vehicle according to the invention, e.g. a car or a truck. The vehicle can be driven manually by a human driver. Alternatively, the vehicle supports semi-autonomous or autonomous driving. Therefore, the driving support system can be a driving support system for autonomously moving the vehicle. In addition, other types of driving assistance systems are known in the art. It is possible for the vehicle to be used to transport occupants, including a driver, or to be used only for the transport of goods.

Objects in the environment of the ego vehicle can be any objects. They can be static objects like houses, road signs, or parked cars. In addition, the objects can move, such as different types of foreign vehicles or pedestrians.

Object shape detection relates in particular to the detection of a representation of the object relative to a ground plane, i. to a width and a length of the corresponding object. Height information of the objects is less important, since the vehicle drives on a ground level and only interacts with the object in this level.

The image sensor is a device that captures and transmits information that is used to provide a two-dimensional data point matrix, which in turn can be represented in a plane. The image sensor converts a variable attenuation of light waves or electromagnetic waves into signals, preferably into electrical signals. Light waves or electromagnetic waves can have different wavelengths. Various types of image sensors can be used depending on the wavelength. For example, a camera can be used for the visible spectrum of the light. Alternatively, image sensors for electromagnetic waves in the infrared range, e.g. at wavelengths of 1000 nm, or in the ultraviolet range, e.g. at wavelengths of 200 nm. Depending on the image sensor, the two-dimensional data point matrix can be provided in Cartesian coordinates or in polar coordinates. The data points themselves can be annotated with several information elements. For example, a color camera delivers a pixel-based image as two-dimensional data, the individual pixels containing information, e.g. in the three color channels RGB. Alternatively, the individual pixels can contain information about the signal intensity and / or a brightness value. Furthermore, the images recorded by the image sensor can be individual still images or image sequences that form videos or films. For example, an image sensor can be an optical sensor, a camera, a thermal imaging device or a night vision sensor.

The second type of environment sensor is an area sensor having an area field of view that at least partially overlaps the image field of view. The area sensor is a device that captures the three-dimensional structure of the environment from the perspective of the environment sensor and usually measures the depth and / or the distance to the closest surfaces. An area sensor can be a LiDAR-based sensor, a radar-based sensor, an infrared-based sensor, or an ultrasonic-based sensor. Radar sensors use radio waves to determine the distance, angle, and / or speed of objects. Ultrasonic sensors work on the principle of reflected sound waves. LiDAR-based sensors measure the distance to an object by irradiating the object with pulsed laser light and measuring reflections of the emitted pulses. Differences in the laser return times, the wavelengths and / or the intensity can then be determined and used to provide the three-dimensional data point matrix that covers at least a part of the surroundings of the vehicle. Infrared sensors also work on the principle of reflected light waves. However, infrared-based sensors are less reliable than the other environmental sensors because their readings tend to fluctuate with varying lighting conditions. The measurements of the distances to the objects can be carried out at individual points in the environment of the ego vehicle across scanning planes, or the measurements can provide a complete image with depth and / or distance measurements at each point in the environment of the ego vehicle. The data determined by the area sensor form a three-dimensional data point matrix that can be provided in spherical coordinates and contain the distance to an object and the position of the object relative to the sensor position, which is determined by the polar and azimuth angles (theta, phi). Alternatively, the data can be determined or converted to Cartesian coordinates, thereby identifying the position of the object relative to the X, Y and Z coordinate axes and the origin of the coordinate system.

The image field of view is a sector or generally a part of the surroundings of the vehicle from which the respective surroundings sensor detects information. The field of view of a sensor can be influenced by the design of the sensor. For example, a camera collects information through a lens that focuses incoming light waves. The curvature of the lens affects the camera's field of view. The field of view of a camera with a fisheye lens is wider than the field of view of a camera with a conventional lens. The image field of view can also be influenced by the size or dimensions of the detector that is used to convert the attenuation of light waves or electromagnetic waves into electrical signals. It is also possible for a sensor to have an image field of view that covers 360 ° of the surroundings of the vehicle. This can be achieved for example by a rotating sensor or by using several interconnected environmental sensors, e.g. through a set of four wide-angle or even fisheye cameras located on four sides of the vehicle. The overlapping of the two fields of view makes it possible to provide information relating to an object at a specific location using both environmental sensors.

The identification of at least one object in the two-dimensional data point matrix relates to the detection of a specific object in the two-dimensional data point matrix. Various techniques are known in the art for identifying the objects, such as image processing.

The identification of at least one object in the three-dimensional data point matrix also relates to the detection of a specific object in the three-dimensional data point matrix. The detection of the object is based, for example, on the detection of a group of data points that are close to one another.

Various approaches for performing a 2D shape estimation of the identified at least one object based on the two-dimensional data point matrix and for performing a 3D shape estimation of the identified at least one object based on the three-dimensional data point matrix are discussed below.

Various methods of merging information are known in the art. Some details regarding the merging of the 2D shape estimate and the 3D shape estimate for the identified at least one object are given below.

The steps of the method can generally be carried out in any suitable order. This implies that some steps e.g. can be run in parallel. For example, the steps required to carry out the 2D shape estimation of the identified at least one object can be carried out in parallel with the steps which are necessary to carry out the 3D shape estimation of the identified at least one object.

Furthermore, the method can generally be carried out with a plurality of environmental sensors of the same type. Therefore, the sensor information of these sensors is processed one by one to carry out respective shape estimates. Based on this, the shape of the identified at least one object can be determined by merging several 2D shape estimates and / or multiple 3D shape estimations are performed for the identified at least one object.

According to a modified embodiment of the invention, the step of performing a 2D shape estimation of the identified at least one object comprises determining a 2D degree of confidence for the 2D shape estimation for the identified at least one object, the step of executing a 3D shape estimation of the identified object at least one object has the determination of a 3D confidence level for the 3D shape estimation for the identified at least one object and comprises the step of determining the shape of the identified at least one object by merging the 2D shape estimation and the 3D shape estimation for the identified at least one Object, the merging of the 2D shape estimation and the 3D shape estimation for the identified at least one object with additional consideration of the 2D degree of confidence and the 3D degree of confidence of the 2D shape estimation and the 3D shape estimation for the identified at least one object. The degree of confidence provides a probability that the respective 2D / 3D shape estimate is correct. Therefore, the degree of confidence lies within the interval [0; 1]. Usually, the confidence level can be viewed as a confidence level with a Gaussian distribution. Based on the degree of confidence, the merging of the 2D shape estimation and the 3D shape estimation can be simplified. The higher the respective degree of confidence, the better the 2D / 3D shape estimation. When determining the shape of the identified at least one object, shape estimates with a higher degree of confidence therefore have a stronger influence on the determination of the shape than shape estimates with a lower degree of confidence.

According to a modified embodiment of the invention, the method has an additional step of determining a degree of confidence for the shape of the identified at least one object, in particular by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object with the 2D degree of confidence and the 3D degree of confidence of the 2D shape estimate and the 3D shape estimate for the identified at least one object. As a result, the shape of the identified at least one object can also be linked to a degree of confidence that can be used when the shape of the identified at least one object is further processed, in particular in a driving assistance system.

According to a modified embodiment of the invention, the method has an additional step of tracking the identified at least one object between different identifications of the at least one object in the two-dimensional data point matrix and / or in the three-dimensional data point matrix, and has the step of performing the 2D shape estimation and / or the 3D shape estimation of the identified at least one object executes the 2D shape estimation and / or the 3D shape estimation taking into account the tracked identifications of the at least one object. Typically, objects move relative to the vehicle over time, either due to movement of the vehicle or due to movement of the object. Therefore, the objects can be seen differently by the image sensor and / or be exposed to different environmental conditions. The more information is available in relation to the identified at least one object, the better the 2D shape estimation and / or the 3D shape estimation. As a result of the movements of the ego vehicle and / or the objects, tracking the identified at least one object enables different 2D shape estimates and / or 3D shape estimates over time to be assigned to the same objects.

According to a modified embodiment of the invention, the step of determining a 2D degree of confidence for the 2D shape estimation and / or of determining a 3D degree of confidence for the 3D shape estimation for the identified at least one object comprises determining a 2D degree of confidence and / or a 3D degree of confidence for a latitude and a longitude of the identified at least one object. The 2D / 3D degree of confidence can be determined independently for latitude and longitude. In other cases, however, a common confidence level can be determined for latitude and longitude. The latitude and longitude are extremely important in determining the shape of the identified at least one object. The width and length are preferably defined in relation to the ego vehicle, i.e. according to the longitudinal and latitudinal directions of the ego vehicle, although other definitions can be used. The only important thing is a difference between different sides of the identified at least one object, e.g. a front and a lateral side. The length and the width can be adjusted if the identified at least one object rotates and faces the ego vehicle different sides during its observation period, e.g. while at least one of the environmental sensors appears in the image field of view.

According to a modified embodiment of the invention, the step of determining a 3D degree of confidence for a latitude and a length of the identified at least one object comprises determining the 3D degree of confidence for the latitude and longitude the length of the identified at least one object based on a position of the identified at least one object in relation to the vehicle, in particular in relation to a position of the area sensor. The position preferably has an angular position of the respective object relative to the vehicle. Depending on the angular position, objects such as third-party vehicles or other rectangular objects face the vehicle with their front / rear or their lateral sides differently. For example, no length information can be determined for an external vehicle in front of the ego vehicle, since the lateral sides do not face the ego vehicle and the room type sensor. Therefore, the confidence level for the width of the other vehicle is very high, while the confidence level for the length is quite low. The same principles apply to an external vehicle in addition to the ego vehicle, in which no width information can be determined because the front / rear of the external vehicle do not face the ego vehicle and the area sensor. Therefore, the degree of confidence for the length of the other vehicle is very high, while the degree of confidence for the width is quite low. The position can also have a distance of the respective object relative to the vehicle. The greater the distance, the lower the number of data points that cover the respective object. This leads to lower levels of confidence for objects that are further away from the ego vehicle.

According to a modified embodiment of the invention, the method has an additional step for converting the 2D degree of confidence for the 2D shape estimation of the identified at least one object, the 3D degree of confidence for the 3D shape estimation of the identified at least one object and / or the degree of confidence for the Form of the identified at least one object in confidence distance information, in particular based on a look-up table. The degree of confidence is a probability that the particular shape of the respective object is correct. Therefore the degree of confidence is a value within the interval [0; 1]. In some cases, however, for easier further processing of the shape of the identified at least one object, it may be preferred to provide the confidence distance information that defines an interval that contains the correct shape with a predetermined probability. In this connection, the shape is regarded as a shape having a Gaussian distribution, for example, and the interval is based on a standard deviation σ from an expected value µ. Therefore, the confidence distance information can be in an interval of e.g. [5 m; 0.1 m]. The look-up table enables easy and quick access to confidence distance information based on the confidence level. The conversion can be defined by an experienced user. In addition, the conversion can e.g. depend on a context.

According to a modified embodiment of the invention, the method has an additional step for executing an object classification for the identified at least one object in the two-dimensional data point matrix, and the step for executing a 2D shape estimation of the identified at least one object comprises executing the 2D shape estimation of the identified at least one object with additional consideration of the classification of the identified at least one object. Since the information from the at least one image sensor does not contain any distance information, the object classification is a suitable means for determining a 2D shape estimate based on the two-dimensional data point matrix that is provided by the at least one image sensor. Sensor information from image sensors is generally highly suitable for object classification. Based on the classification of the identified at least one object, its shape can e.g. can be determined based on shape information which is assigned to objects from the respective class.

According to a modified embodiment of the invention, the step of performing an object classification for the identified at least one object comprises defining a set of object classes and assigning one of the classes to the identified at least one object, each object class being assigned predefined shape information, and has the step of performing a 2D shape estimation of the identified at least one object includes performing the 2D shape estimation of the identified at least one object based on the predefined shape information. Therefore, typical shape information is assigned to each class of objects. This typical shape information can be static information or it can be modified dynamically, e.g. based on a location of the identified at least one object or the like. For example, an average length of cars may be different in different countries. The typical shape information can e.g. can be determined based on statistical data. A simple implementation can be realized based on a look-up table with a set of classes and corresponding shape information.

According to a modified embodiment of the invention, the step of performing an object classification for the identified at least one object comprises determining a probability that the identified at least one object belongs to a class, based on multiple classifications of the identified at least one object. If the same object can be observed over a longer period of time and the method is carried out continuously, this leads to several identifications of identical objects in the vicinity of the vehicle. However, if the identified at least one object or the ego vehicle moves, the visibility of the respective object can change. This relates to different distances of the identified at least one object, different viewing angles, different orientations of the identified at least one object and the ego vehicle relative to one another and so on. In addition, changes in the environmental conditions can lead to changes in the visibility of the identified at least one object. Overall, the identified at least one object will most likely be classified differently during its observation period, ie while it is in the vicinity of the vehicle. Due to the multiple classifications of the respective object, problems in handling incorrect classifications can be overcome.

According to a modified embodiment of the invention, the step of performing an object classification for the identified at least one object includes classifying the identified at least one object in the two-dimensional data point matrix using an image recognition algorithm and / or using a neural network. The processing of a two-dimensional data point matrix from the image sensor, in particular video data that contains a sequence of several individual images per second, is a great challenge. Huge amounts of data have to be processed in real time in order to reliably classify the identified at least one object without delays. However, the resources of the vehicle for processing the data are limited in terms of the space for accommodating processing devices and also in terms of available computing power and electrical power. Even if the technical problems are resolved, resources are limited in terms of price to offer vehicles at an affordable price. A powerful way of processing the two-dimensional data point matrix is to use a neural network. Conventional applications of neural networks for image processing are typically based on deep neural networks. The use of such network types has shown promising results at an affordable price. Neural networks have an input and an output layer and one or more hidden layers. The hidden layers of a deep neural network typically consist of convolutional layers, pooling layers, fully connected layers and normalization layers. The convolutional layers apply a convolution operation to the input and pass the result to the next layer. The convolution emulates the response of a single neuron to visual stimuli. Alternatively or additionally, an image recognition algorithm can be carried out. Image recognition can, for example, be based on genetic algorithms, on approaches based on CAD-like object models such as Primal Sketch, on appearance-based processes such as Edge Matching or Conquer And Search, or on other feature-based processes such as Histogram of Oriented Gradients (HOG), Haar-like Features, Scale-Invariant Feature Transformation (SIFT) or Speeded UP Robust Feature (SURF).

Similarly, the step of identifying the at least one object in the two-dimensional data point matrix can already include identifying the at least one object in the two-dimensional data point matrix using an image recognition algorithm and / or using a neural network. The above principles also apply in this case.

According to a modified embodiment of the invention, the step of determining the shape of the identified at least one object by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object includes executing a Bayesian merging algorithm. However, other algorithms for merging the 2D shape estimation and the 3D shape estimation can also be used. Furthermore, the same algorithm can be used to e.g. provide a degree of confidence for the shape of each object.

According to a modified embodiment of the invention, the step of providing a two-dimensional data point matrix, which represents the surroundings of the vehicle in the image field of view of the at least one image sensor, comprises providing the two-dimensional data point matrix by at least one camera. In conventional vehicles, one or more cameras for monitoring the surroundings are often already integrated. In this case, the use of a camera as an image sensor does not add any additional costs when the method is used. Furthermore, cameras with special lenses, for example with fisheye lenses, can be used, which have a wide field of view of up to 180 degrees or even more. Cameras provide precise information about the type of object. By using night vision technology, in which parts of the electromagnetic spectrum that is invisible to humans, such as near-infrared or ultraviolet radiation, using very sensitive detectors, can capture information about objects by the camera in situations where a human driver is restricted.

According to a modified embodiment of the invention, the step of providing a three-dimensional data point matrix, which represents the surroundings of the vehicle in the area field of view, by means of at least one area sensor, comprises providing the three-dimensional data point matrix by means of at least one LiDAR-based sensor and / or by at least one radar sensor. Many vehicles use a LiDAR based sensor and / or a radar sensor, e.g. in collision avoidance systems, for example to determine a distance to another vehicle in front of the ego vehicle. LiDAR-based sensors and / or radar sensors provide precise information about the position of an object, in particular about its distance from the ego vehicle. LiDAR based sensors and / or radar sensors can have fields of view of up to 360 degrees, e.g. using rotating sensors. It is also possible to use several area sensors, e.g. a combination of a LiDAR-based sensor and a radar sensor.

These and other aspects of the invention will be apparent and illustrated in the embodiments described below. Individual features that are shown in the embodiments can form an aspect of the present invention on their own or in combination. Features of the various embodiments can be transferred from one embodiment to another embodiment.

• 1 eine schematische Ansicht eines Fahrzeugs gemäß einer ersten bevorzugten Ausführungsform mit einem Fahrunterstützungssystem mit zwei Typen von Umgebungssensoren, einschließlich eines Bereichssensors und eines Bildsensors, und mit einer Verarbeitungseinheit in einer Umgebung mit mehreren Objekten;
• 2 eine schematische Ansicht des Fahrzeugs von 1 mit vier Bereichssichtfeldern, die eine Rundumsicht des Bereichssensors liefern, zusammen mit einer Konfidenzabschätzung für das hinter dem Ego-Fahrzeug befindliche identifizierte Objekt;
• 3 eine schematische Ansicht des Fahrzeugs von 2 mit einer Konfidenzabschätzung für das identifizierte Objekt, wobei sich das identifizierte Objekt neben dem Ego-Fahrzeug befindet;
• 4 ein Diagramm zum Darstellen eines Konfidenzgrades für eine 2D-Formabschätzung basierend auf einer Verfolgung von Objekten über die Zeit und dem Ausführen einer Klassifizierung der Objekte;
• 5 ein Diagramm zum Darstellen des Konfidenzgrades für eine 2D-Formabschätzung für mehrere Objekte basierend auf einer Klassifizierung der Objekte mit einer Umwandlung in Konfidenzabstandsinformation;
• 6 ein Diagramm zum Darstellen einer Verschmelzung einer 2D-Formabschätzung und einer 3D-Formabschätzung für zwei Objekte basierend auf einer Klassifizierung der Objekte mit einer Umwandlung in Konfidenzabstandsinformation; und
• 7 ein Diagramm zum Darstellen eines Konfidenzgrades für eine 2D-Formabschätzung für mehrere Objekte basierend auf einer Klassifizierung der Objekte mit einer Umwandlung in Konfidenzabstandsinformation.
• 1 zeigt ein Fahrzeug 10 mit einem Fahrunterstützungssystem 12 gemäß einer ersten bevorzugten Ausführungsform.

Show it:

• 1 a schematic view of a vehicle according to a first preferred embodiment with a driving support system with two types of environment sensors, including a range sensor and an image sensor, and with a processing unit in an environment with a plurality of objects;
• 2 a schematic view of the vehicle of FIG 1 with four area fields of view that provide an all-round view of the area sensor, together with a confidence estimate for the identified object located behind the ego vehicle;
• 3 a schematic view of the vehicle of FIG 2 with a confidence estimate for the identified object, the identified object being adjacent to the ego vehicle;
• 4th a diagram showing a degree of confidence for a 2D shape estimation based on tracking objects over time and performing classification of the objects;
• 5 a diagram for illustrating the degree of confidence for a 2D shape estimation for a plurality of objects based on a classification of the objects with a conversion into confidence distance information;
• 6th a diagram for illustrating a merging of a 2D shape estimation and a 3D shape estimation for two objects based on a classification of the objects with a conversion into confidence distance information; and
• 7th a diagram showing a degree of confidence for a 2D shape estimation for a plurality of objects based on a classification of the objects with a conversion to confidence distance information.
• 1 shows a vehicle 10 with a driving assistance system 12 according to a first preferred embodiment.

The vehicle 10 , also known as an ego vehicle, can be any type of vehicle 10 be, e.g. a car or a truck. The vehicle 10 can be driven manually by a human driver. Alternatively, the vehicle provides support 10 semi-autonomous or autonomous driving. Therefore, the driving support system 12 a driving assistance system 12 for autonomous movement of the vehicle 10 be. In addition, other types of driving assistance systems are in the art 12 known. It is possible that the vehicle 10 Occupants, including a driver, are transported or used only for the transport of goods.

The driving support system 12 is for object shape detection based on two types of environmental sensors 14th , 16 intended. As a first type of environmental sensor 14th , 16 instructs the driving assistance system 12 an image sensor 14th to provide a two-dimensional data point matrix, which the environment 18th of the vehicle 10 in an image field of view 20th represents. As a second type of environmental sensor 14th , 16 instructs the driving assistance system 12 an area sensor 16 to provide a three-dimensional data point matrix, which the environment 18th of the vehicle 10 in an area field of view 22nd represents. The area field of view 22nd according to the first embodiment, covers an area of 360 ° of the surroundings 18th of the vehicle 10 and is therefore indicated by a spiral arrow. The area field of view 22nd , according to the first embodiment, the range of 360 ° of Surroundings 18th of the vehicle 10 covers is obtained by four individual LiDAR devices, not shown individually and which are located on each side of the vehicle 10 are located, each of these facilities having a partial field of view 30th of about 150 °. The partial fields of view 30th overlap, such as in the 2 and 3 can be seen, and together form the area field of view 22nd that covers the area of 360 °. Sensor information of the LiDAR devices is processed together so that the LiDAR devices together form the area sensor 16 form.

The image sensor 14th is a device that collects and transmits information that is used to provide a two-dimensional data point matrix that can in turn be represented in a plane. The image sensor 14th converts a variable attenuation of light waves or electromagnetic waves into electrical signals. The light waves or electromagnetic waves can have different wavelengths. Different types of image sensors can be used depending on the wavelength 14th be used. For example, a camera can be used for the visible spectrum of the light. The two-dimensional data point matrix can be provided in Cartesian coordinates or in polar coordinates. The data points themselves can be annotated with several information elements. For example, a color camera delivers a pixel-based image as a two-dimensional data point matrix, with the individual pixels containing information, for example in the three color channels RGB. Alternatively, the individual pixels can contain information about the signal intensity and / or a brightness value. In this embodiment, the image sensor is 14th an optical front camera.

The area sensor 16 is a device that shows the three-dimensional structure of the environment from the perspective of the environment sensor 14th , 16 and usually measures the depth and / or the distance to the closest surfaces. In this embodiment the area sensor is 16 a sensor based on LiDAR. In an alternative embodiment, the area sensor is 16 a radar-based sensor, an infrared-based sensor, or an ultrasonic-based sensor. The LiDAR-based sensor measures the distance to an object 24 by illuminating the object 24 with pulsed laser light and measuring reflections of the emitted pulses. Differences in laser return times, wavelengths, and / or intensity are then determined and used to provide the three-dimensional data point matrix representing the environment 18th of the vehicle 10 covers. The measurements of the distance to the objects 24 can at individual points in the area 18th of the ego vehicle 10 across scan planes, or the measurements can be performed at any point in the environment 18th of the ego vehicle 10 provide a complete picture with depth and / or distance readings. The by the area sensor 16 certain data can be provided in spherical coordinates representing the distance to an object 24 and the position of the object 24 relative to the sensor position, which is determined by the polar and azimuth angles (theta, phi). Alternatively, the data can be determined or converted to Cartesian coordinates and can include the position of the object 24 relative to the coordinate axes X, Y and Z and the origin of the coordinate system.

The objects 24 in the neighborhood 18th of the ego vehicle 10 can use any type of object 24 be. You can use static objects 24 be like houses, traffic signs, or parked cars. In addition, the objects can 24 move, such as different types of foreign vehicles or pedestrians.

The field of vision 20th , 22nd is a sector of the area 18th of the vehicle 10 from which the respective environmental sensor 14th , 16 Sensor information captured. The field of vision 20th , 22nd of the environmental sensors 14th , 16 is determined by the design of the respective environmental sensor 14th , 16 influenced. For example, the front camera records information using a lens that focuses incoming light waves. The curvature of the lens affects the field of view 20th the camera.

The driving support system 12 also has a control unit 26th on that via a communication bus 28 with the two environmental sensors 14th , 16 connected is. The control unit 26th receives the two-dimensional data point matrix, which the environment 18th of the vehicle 10 as well as the three-dimensional data point matrix that represents the surroundings 18th of the vehicle 10 represents, via the communication bus 28 from the two environmental sensors 14th , 16 .

A method for object shape detection that is implemented by the driving assistance system is discussed below 12 of the first embodiment is carried out. A flow chart of the process is shown in 7th shown. The object shape detection is based on the two types of environmental sensors 14th , 16 that are in the driving assistance system 12 of the vehicle 10 of the first embodiment can be used. The procedure is described with further reference to the 2 to 6th described.

The procedure begins with step S100 , which refers to providing a two-dimensional data point matrix that represents the environment 18th of the vehicle 10 in the field of view 20th of the image sensor 14th represents. The two-dimensional data point matrix is from the image sensor 14th via the communication bus 28 to the control unit 26th which then carries out further processing.

step S110 refers to identifying at least one object 24 in the two-dimensional data point matrix. Identifying the at least one object 24 in the two-dimensional data point matrix refers to the detection of a particular object 24 . Various methods are known in the art for performing such as image processing to identify the objects 24 known. For example, identifying the objects 24 in the two-dimensional data point matrix using an image recognition algorithm and / or using a neural network. The following apply with reference to step S120 principles discussed in more detail.

step S120 refers to performing an object classification on the identified at least one object 24 in the two-dimensional data point matrix. step S120 is carried out using an image recognition algorithm and / or using a neural network. For example, the neural network for image processing in this embodiment is based on a deep neural network. The neural network has an input and an output layer and one or more hidden layers. The hidden layers of the deep neural network consist of convolutional layers, pooling layers, fully connected layers and normalization layers. The convolutional layers apply a convolution operation to the input, with the result being passed to the next layer. The convolution emulates the response of a single neuron to visual stimuli. Alternatively, an image recognition algorithm can be carried out. Image recognition is based, for example, on genetic algorithms, on approaches based on CAD-like object models such as Primal Sketch, on appearance-based methods such as Edge Matching or Conquer And Search, or on other feature-based methods such as Histogram Oriented Gradients (HOG), Haar -like Features, Scale-Invariant Feature Transformation (SIFT) or Speeded UP Robust Feature (SURF).

The object classification for the identified at least one object 24 involves defining a set of classes of objects 24 and assigning one of the classes to the identified at least one object 24 . Any class of objects 24 a predefined shape information is assigned, and the step of performing a 2D shape estimation of the identified at least one object 24 instructs performing the 2D shape estimation of the identified at least one object 24 based on the predefined shape information. Hence every class of objects 24 typical shape information for objects 24 assigned to this class. As in the 4th and 5 As can be seen, the identified can be at least one object 24 classified as unknown, pedestrian, bicycle, car, truck, animal, barrier or virtual.

step S130 refers to tracking the identified at least one object 24 between different identifications of the at least one object 24 in the two-dimensional data point matrix. Hence the identified at least one object 24 tracked as long as it is at least within the field of view 20th or the area field of view 22nd is located.

step S140 refers to performing a 2D shape estimation of the identified at least one object 24 based on the two-dimensional data point matrix. The 2D shape estimate of the identified at least one object 24 is taking into account the classification of the identified at least one object 24 and the tracked identifications of the at least one object 24 executed.

As referring to 4th can be seen, every object 24 initialized on a first detection with a count of zero indicating a number of classifications of each class. Furthermore, each class has the same probability of representing the identified at least one object 24 . This happens with the observation period 0 , for example before the first classification of the at least one object 24 he follows. Every time that identified at least one object 24 is identified and classified again, the corresponding count is incremented and the probability of each class for the representation of the identified at least one object 24 customized. As in 4th it can be seen is if the identified at least one object 24 For example, if an observation period of 42 is reached, the identified at least one object 24 classified as unknown, cars and trucks. The highest number of classifications of the identified at least one object 24 refers to the truck class, so this class is assigned the highest probability. Hence the probability that the identified at least one object 24 belongs to a class based on multiple classifications of the identified at least one object 24 certainly.

The 2D shape estimate of the identified at least one object 24 is based on the determined probability of belonging to the identified at least one object 24 for the various classes along with a look-up table giving typical shape information. For example, according to this embodiment, the class car is assigned a length of 4.5 meters, the class truck is assigned a length of 12 meters, the class motorcycle is assigned a length of 1.5 meters and the class pedestrian is assigned a length of 1 Meters assigned. The width of the identified at least one object is also similar 24 certainly.

Furthermore, a 2D degree of confidence is used for the 2D shape estimation for the identified at least one object 24 certainly. The 2D degree of confidence in relation to the length of the identified at least one object 24 is based on the determined probability of belonging to the identified at least one object 24 to different classes along with the look-up table, which also gives typical variance information. For example, according to this embodiment, the class car is assigned a variance of 1.5 meters, the class truck is assigned a variance of 6 meters, the class motorcycle is assigned a variance of 0.5 meters and the class pedestrian is assigned a variance of 0 , Assigned 5 meters. The 2D degree of confidence with regard to the width of the identified at least one object is also similar 24 certainly.

Therefore, in this embodiment, the degree of confidence for the 2D shape estimation for a latitude and a longitude of the identified at least one object 24 certainly. The width and length of the object 24 are preferably related to the ego vehicle 10 defined, ie according to the length and width direction of the ego vehicle 10 although other definitions can be used. It is only essential to distinguish between different sides of the identified at least one object 24 , e.g. a front or a back and a lateral side. The length and width can be adjusted if the identified at least one object 24 turns and the ego vehicle 10 turns to different sides during his observation period, eg while at least one image sensor is in the field of view 14th is located.

Three examples of identified objects 24 and their estimated shape in terms of length are in 3 with the left, middle and right vectors being the left, middle and right objects 24 in 5 correspond. Each vector gives the probability for each class according to 4th on. Every object 24 is shown along with its resulting length and the 2D confidence level for the length.

step S150 refers to providing a three-dimensional data point matrix that represents the environment 18th of the vehicle 10 in the area field of view of the area sensor 16 represents. The three-dimensional data point matrix is provided by the area sensor 16 via the communication bus 28 to the control unit 26th which then carries out further processing.

step S160 refers to identifying at least one object 24 in the three-dimensional data point matrix. Identifying at least one object 24 in the three-dimensional data point matrix, the acquisition of a specific object 24 in the three-dimensional data point matrix. Detecting the at least one object 24 for example, is based on acquiring a group of data points that are close together.

step S170 refers to performing a 3D shape estimation of the identified at least one object 24 based on the three-dimensional data point matrix. This includes determining a length and a width of the identified at least one object 24 based on the data points of the three-dimensional matrix. Therefore, depending on an orientation of the object relative to the vehicle 10 Data points based on reflections from a lateral side of the identified at least one object 24 evaluated to a length of the identified at least one object 24 to determine, and data points based on reflections from a front or a back of the identified at least one object 24 are evaluated to a width of the identified at least one object 24 to determine.

Furthermore, a 3D degree of confidence is used for the 3D shape estimation for the identified at least one object 24 certainly. This includes the determination of a 3D degree of confidence for a latitude and a length of the identified at least one object 24 based on a position of the identified at least one object 24 in relation to the vehicle 10 , particularly with respect to a position of the area sensor 16 at the vehicle 10 . The position preferably includes an angular position of the respective object 24 relative to the vehicle 10 . Objects turn depending on the angular position 24 like foreign vehicles or other rectangular objects their front / rear or their lateral sides to the vehicle 10 different to. Details are in the 2 and 3 evident. For example, for an object 24 in front of the ego vehicle 10 no length information can be determined because the lateral sides of the object 24 the ego vehicle 10 and the area sensor 16 are not facing. Hence the confidence level for the width is this Object 24 very high, whereas the degree of confidence for the length of this object 24 is pretty low. The same principles apply to an object 24 next to the ego vehicle 10 where width information cannot be determined because the front / back of the object 24 the ego vehicle 10 and the area sensor 16 is not facing. Hence the confidence level for the length of the object 24 very high, while the latitude confidence level is quite low. The position can also be a distance of the corresponding object 24 relative to the vehicle 10 include. The greater the distance, the fewer the number of data points the corresponding object is 24 cover. This leads to lower confidence levels for objects 24 that continues from the ego vehicle 10 are away.

The degree of confidence for longitude and latitude is determined based on a Gaussian curve, which is mapped to angular directions from 0 ° to 180 ° or + 90 ° to -90 °. The degree of confidence refers to the height if the curve is from the angular position of the at least one object 24 depends.

Thus, in this embodiment, the degree of confidence for the 3D shape estimation for a width and a length of the identified at least one object 24 certainly. The width and length of the object 24 are preferably related to the ego vehicle 10 defined, ie according to the length and width direction of the ego vehicle 10 although other definitions can be used. It is only essential to distinguish between different sides of the identified at least one object 24 , e.g. a front and a lateral side. The length and width can be adjusted if the identified at least one object 24 turns and the ego vehicle 10 turns to different sides during its observation period, eg while at least one of the area sensors is in the field of view 16 is located.

step S180 refers to converting the 3D confidence level for the 3D shape estimate for the identified at least one object 24 in confidence distance information, in particular based on a look-up table. The degree of confidence is a probability that the particular shape of the corresponding object 24 correct is. Therefore the degree of confidence is a value within the interval [0; 1]. The confidence distance information defines an interval that contains the correct shape with a predetermined probability. In this context, the shape is viewed as a Gaussian distribution and the interval is based on a standard deviation σ of the expected value µ. Therefore, the confidence distance information can be within an interval of, for example, [5 m; 0.1 m]. The look-up table enables easy and quick access to confidence distance information based on the confidence level. The conversion is defined by an experienced user.

As above with reference to step S140 discussed, is the 2D confidence level for the 2D shape estimate for the identified at least one object 24 already provided as confidence distance information.

step S190 relates to determining the shape of the identified at least one object 24 by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object 24 . Hence, as in 6th it can be seen the shape of the identified at least one object 24 determined with its length and its width relative to a ground plane. A Bayesian blend algorithm is used to perform the blend of the two shape estimates. In alternative embodiments, other algorithms are used to merge the 2D shape estimation and the 3D shape estimation.

The merging of the 2D shape estimation and the 3D shape estimation for the identified at least one object 24 is also taking into account the 2D degree of confidence and the 3D degree of confidence of the 2D shape estimation and the 3D shape estimation for the identified at least one object 24 executed. The higher the respective degree of confidence, the better the 2D / 3D shape estimation. If the shape of the identified at least one object 24 is determined, shape estimates with a higher degree of confidence therefore have a stronger influence on the determination of the shape than shape estimates with a lower degree of confidence.

step S200 relates to determining a degree of confidence for the shape of the identified at least one object 24 in particular by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object 24 with the 2D degree of confidence and the 3D degree of confidence of the 2D shape estimation and the 3D shape estimation for the identified at least one object 24 . As a result, the shape of the identified at least one object can also be 24 be provided with a degree of confidence that is used in the further processing of the shape of the identified at least one object 24 can be used, in particular in a further driving assistance system.

The steps of the method can generally be carried out in any suitable order. This means that some steps can be carried out in parallel, for example. For example, the for performing the 2D shape estimation of the identified at least one object 24 necessary steps are carried out in parallel to the steps required to carry out the 3D shape estimation of the identified at least one object 24 required are.

List of reference symbols

10

vehicle

12th

Driving support system

14th

Environment sensor, image sensor

16

Environment sensor, area sensor

18th

Surroundings

20th

Field of view

22nd

Area field of view

24

object

26th

Control unit

28

Communication bus

30th

Partial field of view

Claims (15)

Method for object shape detection based on two types of environment sensors (14, 16) which are used in a driving support system (12) of a vehicle (10), the first type of environment sensor (14, 16) being an image sensor (14) with an image field of view (20) and the second type of environment sensor is an area sensor (16) having an area field of view (22) which at least partially overlaps the image field of view (20), comprising the steps of: Providing a two-dimensional data point matrix, which represents the surroundings (18) of the vehicle (10) in the image field of view (20), using at least one image sensor (14); Identifying at least one object (24) in the two-dimensional data point matrix; Performing a 2D shape estimation of the identified at least one object (24) based on the two-dimensional data point matrix; Providing a three-dimensional data point matrix which represents the surroundings (18) of the vehicle (10) in the area field of view (22) using at least one area sensor (16); Identifying at least one object (24) in the three-dimensional data point matrix; Performing a 3D shape estimation of the identified at least one object (24) based on the three-dimensional data point matrix; and Determining the shape of the identified at least one object (24) by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object (24). Procedure according to Claim 1 characterized in that the step of performing a 2D shape estimate of the identified at least one object (24) comprises determining a 2D degree of confidence for the 2D shape estimate for the identified at least one object (24); the step of performing a 3D shape estimation of the identified at least one object (24) comprises determining a 3D degree of confidence for the 3D shape estimation for the identified at least one object (24); and the step of determining the shape of the identified at least one object (24) by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object (24) merging the 2D shape estimation and the 3D shape estimation for the identified at least one an object (24), taking into account the 2D degree of confidence and the 3D degree of confidence of the 2D shape estimation and the 3D shape estimation for the identified at least one object (24). Procedure according to Claim 2 , characterized in that the method includes an additional step for determining a degree of confidence for the shape of the identified at least one object (24), in particular by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object (24) with the 2D And the 3D confidence level of the 2D shape estimation and the 3D shape estimation for the identified at least one object (24). Method according to one of the preceding Claims 2 or 3 characterized in that the method comprises an additional step of tracking the identified at least one object (24) between different identifications of the at least one object (24) in the two-dimensional data point matrix and / or in the three-dimensional data point matrix; and the step of executing the 2D shape estimation and / or the 3D shape estimation of the identified at least one object (24) comprises executing the 2D shape estimation and / or the 3D shape estimation taking into account the tracked identifications of the at least one object (24) . Method according to one of the preceding Claims 2 to 4th , characterized in that the step of determining a 2D degree of confidence for the 2D shape estimation and / or for determining a 3D degree of confidence for the 3D shape estimation for the identified at least one object (24) includes determining a 2D degree of confidence and / or a 3D degree of confidence for a width and has a length of the identified at least one object (24). Procedure according to Claim 5 , characterized in that the step of determining a 3D degree of confidence for a latitude and a length of the identified at least one object (24) includes determining the 3D degree of confidence for the latitude and the longitude of the identified at least one object (24) based on a Position of the identified at least one object (24) in relation to the vehicle (10), in particular in relation to a position of the area sensor (16). Method according to one of the Claims 2 to 6th , characterized in that the method includes an additional step for converting the 2D degree of confidence for the 2D shape estimation for the identified at least one object (24), the 3D degree of confidence for the 3D shape estimation for the identified at least one object (24) and / or the degree of confidence for the shape of the identified at least one object (24) in confidence distance information, in particular based on a look-up table. Method according to one of the preceding claims, characterized in that the method has an additional step for carrying out an object classification for the identified at least one object (24) in the two-dimensional data point matrix; and the step of performing a 2D shape estimation of the identified at least one object (24) comprises executing the 2D shape estimation of the identified at least one object (24) with additional consideration of the classification of the identified at least one object (24). Procedure according to Claim 8 characterized in that the step of performing an object classification for the identified at least one object (24) comprises defining a set of object classes (24) and assigning one of the classes to the identified at least one object (24), each object class having one predefined shape information is assigned; and the step of performing a 2D shape estimation of the identified at least one object (24) comprises performing the 2D shape estimation of the identified at least one object (24) based on the predefined shape information. Procedure according to Claim 8 or 9 characterized in that the step of performing an object classification for the identified at least one object (24) is determining a probability of the at least one identified object (24) belonging to a specific class based on a plurality of classifications of the identified at least one object (24 ) having. Method according to one of the preceding Claims 8 to 10 , characterized in that the step of performing an object classification for the identified at least one object comprises classifying the identified at least one object (24) in the two-dimensional data point matrix (26) using an image recognition algorithm and / or using a neural network. A method according to any one of the preceding claims, characterized in that the step of determining the shape of the identified at least one object (24) by merging the 2D shape estimation and the 3D shape estimation for the identified at least one object (24) is executing a Bayesian merging algorithm having. Method according to one of the preceding claims, characterized in that the step of providing a two-dimensional data point matrix (26), which represents the surroundings of the vehicle (10) in the image field of view (18), by means of at least one image sensor (14) is providing the two-dimensional data point matrix ( 26) has at least one camera (15). Method according to one of the preceding claims, characterized in that the step of providing a three-dimensional data point matrix (28), which represents the surroundings of the vehicle (10) in the area field of view (20), by means of at least one area sensor (16) is providing the three-dimensional data point matrix ( 28) has at least one LiDAR-based sensor (17) and / or at least one radar sensor. Driving support system (12) for object shape detection based on two types of environment sensors according to the method according to one of the preceding methods claims 1 to 14th wherein the driving support system (12) comprises: at least one image sensor (14) as a first type of environment sensor for providing a two-dimensional data point matrix (26) which represents the environment of the vehicle (10) in an image field of view (18); and at least one area sensor (16) as a second type of environment sensor for providing a three-dimensional data point matrix (28) which represents the environment of the vehicle (10) in an area field of view (20), wherein the image field of view (20) at least partially overlaps the area field of view (18).

Images