DE102020105701A1 - Method and system for generating a robust feature descriptor - Google Patents

Abstract

The present disclosure relates to a method for generating a robust feature descriptor, comprising: acquiring two-dimensional image data by a first sensor system; acquisition of depth measurement data by the first sensor system or a second sensor system; and combining the two-dimensional image data and the depth measurement data to generate a feature descriptor.

Description

The present disclosure relates to a method for generating a robust feature descriptor, a driver assistance method, a storage medium for carrying out the method, a system for generating a robust feature descriptor and a vehicle with the system. The present disclosure relates in particular to the detection of dynamics in a three-dimensional environment for use in driver assistance systems to increase the reliability and safety of the driver assistance systems, such as, for example, driver assistance systems for (partially) autonomous driving.

State of the art

The development of driver assistance functions, for example for (partially) autonomous driving, is becoming increasingly important. In order to enable driver assistance systems to perceive the environment as precisely as possible, sensor measurements from various sensor types, such as cameras, LiDAR and radars, can be used. Each type of sensor is subject to different physical measuring principles. In reality, the vehicle environment consists of many different objects / subjects such as different vehicle types, people and animals, static objects such as garbage cans, traffic control systems, natural objects and terrain. Since the perception of all these objects / subjects is also dependent on the measuring principle used, it is difficult to ensure a precise perception of the surroundings.

Disclosure of the invention

It is an object of the present disclosure to indicate a method for generating a robust feature descriptor, a driver assistance method, a storage medium for carrying out the method, a system for generating a robust feature descriptor and a vehicle with the system that is capable of generating a robust feature. Descriptors enable, for example, to precisely model dynamics in a vehicle environment. In particular, it is an object of the present disclosure to increase the reliability of driver assistance systems by detecting the surroundings.

These objects are achieved by the subject matter of the independent claims. Advantageous refinements are given in the subclaims.

According to an independent aspect of the present disclosure, a method for generating a robust feature descriptor is provided. The method comprises capturing two-dimensional image data by a first sensor system; acquisition of depth measurement data by the first sensor system or a second sensor system; and combining the two-dimensional image data and the depth measurement data to generate a feature descriptor based on the combination.

The acquisition of two-dimensional image data and the acquisition of depth measurement data can take place either one after the other or at the same time.

According to the invention, a two-dimensional image (e.g. an RGB image) and depth information are combined in order to determine a (e.g. CCN-based) feature descriptor. In this way, in particular, improved accuracy in the case of pixel-by-pixel matching for the flow estimation can be made possible.

In some embodiments, the two-dimensional image data and the depth measurement data are acquired by different sensor systems, i.e. the first sensor system and the second sensor system, respectively. Alternatively, the method includes the simultaneous acquisition of image and depth measurement data by a single sensor system (e.g. RGB-D camera).

The method preferably uses a neural network, and in particular a convolutional neural network, CNN. CNN-based solutions result in higher accuracy compared to handcrafted systems.

The two-dimensional image data and the depth measurement data preferably share the same receptive field. This is made possible in particular by the early fusion of the two-dimensional image data and the depth measurement data.

The method preferably further comprises training the neural network using a triplet-based network, with corresponding pixels in the feature maps being encoded in such a way that they are closer in the mapping space than non-corresponding pixels.

The combining of the two-dimensional image data and the depth measurement data preferably comprises a linking of the depth as Cartesian 3D coordinates with an RGB image.

According to a further independent aspect of the present disclosure, a driver assistance method for a vehicle, in particular a motor vehicle, is specified. The driver assistance method comprises activating a driving function for automated driving using the robust feature descriptor generated with the method described above. In particular, the Feature descriptors estimate the dense 3D field of motion of a scene that moves in whole or in part with respect to, for example, a camera.

According to a further aspect, a software (SW) program is described. The SW program can be set up to be executed on one or more processors, and thereby to execute the method described in this document for generating a robust feature descriptor and / or the driver assistance method for a vehicle.

According to a further aspect, a storage medium is described. The storage medium can comprise a SW program which is set up to be executed on one or more processors and thereby to execute the method described in this document for generating a robust feature descriptor and / or the driver assistance method for a vehicle.

According to a further independent aspect of the present disclosure, a system for generating a robust feature descriptor is provided. The system comprises at least one processor unit which is set up to: receive two-dimensional image data that are acquired by a first sensor system; Receive depth measurement data acquired by the first sensor system or a second sensor system; and combine the two-dimensional image data and the depth measurement data to generate a feature descriptor based on the combination.

In particular, the system is set up to carry out the method described in this document for generating a robust feature descriptor.

According to a further independent aspect of the present disclosure, a vehicle, in particular a motor vehicle, is specified. The vehicle comprises the system for generating a robust feature descriptor according to the embodiments described in this document.

The term vehicle includes cars, trucks, buses, mobile homes, motorcycles, etc., which are used to transport people, goods, etc. In particular, the term includes motor vehicles for passenger transport.

The vehicle preferably comprises the first sensor system and the second sensor system. The first sensor system and the second sensor system can be different sensor systems, i.e. the first sensor system and the second sensor system can use different measuring principles.

The first sensor system can include one or more cameras, and in particular image cameras. The second sensor system can include one or more 3D sensors, such as one or more LiDAR sensors.

The vehicle preferably comprises a driver assistance system, such as a driver assistance system for automated (partially autonomous or autonomous) driving. The driver assistance system uses the robust feature descriptor according to the embodiments described in this document to carry out the automated driving.

In the context of the document, the term “automated driving” can be understood to mean driving with automated longitudinal or lateral guidance or autonomous driving with automated longitudinal and lateral guidance. The automated driving can be, for example, driving on the motorway for a longer period of time or driving for a limited time as part of parking or maneuvering. The term “automated driving” includes automated driving with any degree of automation. Exemplary degrees of automation are assisted, partially automated, highly automated or fully automated driving. These degrees of automation were defined by the Federal Highway Research Institute (BASt) (see BASt publication “Research compact”, edition 11/2012).

With assisted driving, the driver continuously performs the longitudinal or lateral guidance, while the system takes on the other function within certain limits. With partially automated driving (TAF), the system takes over the longitudinal and lateral guidance for a certain period of time and / or in specific situations, whereby the driver has to constantly monitor the system as with assisted driving. With highly automated driving (HAF), the system takes over the longitudinal and lateral guidance for a certain period of time without the driver having to permanently monitor the system; however, the driver must be able to take control of the vehicle within a certain period of time. With fully automated driving (VAF), the system can automatically cope with driving in all situations for a specific application; a driver is no longer required for this application.

The four degrees of automation mentioned above correspond to SAE levels 1 to 4 of the SAE J3016 standard (SAE - Society of Automotive Engineering). For example, highly automated driving (HAF) Level 3 corresponds to the SAE J3016 standard. Furthermore, SAE level 5 is still in the SAE J3016 as highest degree of automation provided, which is not included in the definition of the BASt. SAE level 5 corresponds to driverless driving, in which the system can automatically cope with all situations like a human driver during the entire journey; a driver is generally no longer required.

Figure list

• 1 ein Flussdiagramm eines Verfahrens zum Erzeugen eines robusten Feature-Deskriptors gemäß Ausführungsformen der vorliegenden Offenbarung,
• 2 eine deepRGBXYZ-Architektur gemäß Ausführungsformen der vorliegenden Offenbarung,
• 3 ein Triplett-basiertes Netzwerk gemäß Ausführungsformen der vorliegenden Offenbarung, und
• 4 ein Fahrzeug mit einem Fahrassistenzsystem zum automatisierten Fahren gemäß den Ausführungsformen der vorliegenden Offenbarung.

Exemplary embodiments of the disclosure are shown in the figures and are described in more detail below. Show it:

• 1 a flowchart of a method for generating a robust feature descriptor according to embodiments of the present disclosure,
• 2 a deepRGBXYZ architecture in accordance with embodiments of the present disclosure,
• 3 a triplet-based network in accordance with embodiments of the present disclosure, and
• 4th a vehicle with a driver assistance system for automated driving according to the embodiments of the present disclosure.

Embodiments of the disclosure

A robust perception of a vehicle environment is an essential task for reliable autonomous driving functions. To achieve this goal, dynamic changes in the environment must be recorded. For example, the so-called dense scene flow estimation can be used for this purpose. With the help of the scene flow estimation, a 3D geometry and the 3D motion field can be calculated so that a comprehensive representation of a dynamic environment is made possible. The scene flow can, for example, be calculated based on dense pixel matching in stereo images. This means that adequate quality can be achieved in many scenarios.

Pixel-by-pixel matching provides a basis for many tasks in computer vision, such as image retrieval, object recognition and flow estimation. Particularly in the case of flow estimation, there is a challenge in aligning scenes with dynamic objects. Robust local feature descriptors play an important role here in order to find dense, accurate matches by comparing a distance from local descriptors (i.e. feature maps).

In recent years, handcrafted descriptors have been used in many applications, such as SIFT, DAISY, and HOG. In addition, unmistakable and robust descriptors can be calculated through deep neural networks. Here, however, an accuracy can be reduced due to a receptive field used in the calculation of the descriptors.

The present disclosure proposes a new approach to determining a robust feature descriptor that enables improved accuracy in matching compared to the approaches mentioned above. In the inventive approach, a two-dimensional image (e.g., an RGB image) and depth information are combined to determine a CCN-based feature descriptor with a large receptive field. In this way, in particular, improved accuracy in the case of pixel-by-pixel matching for the flow estimation can be made possible.

1 shows a flow diagram of a method 100 for generating a robust feature descriptor in accordance with embodiments of the present disclosure. The procedure 100 can be implemented by appropriate software that can be executed by one or more processors (e.g. a CPU).

The procedure 100 includes in the block 110 acquiring two-dimensional image data by a first sensor system; in the block 120 acquisition of depth measurement data by the first sensor system or a second sensor system; and in the block 130 combining the two-dimensional image data and the depth measurement data to generate a (dense) feature descriptor.

The first sensor system can include one or more cameras, and in particular image cameras. The second sensor system can include one or more 3D sensors, such as one or more LiDAR sensors.

The dense feature descriptor according to the invention is explained in detail below using an exemplary embodiment.

The approach according to the invention is based on the combination of RGB images and depth information about the image area. The 3D sensors of the second sensor system can “perceive” a depth map, for example using pattern projection, as a flight time or as a point cloud (LiDAR sensors).

If the image data and the depth are recorded by two different sensors or sensor systems, a synchronization and a Calibration may be required. In addition, an intrinsic calibration, for example the main points of a 2D image sensor and a focal length, is generally known. This enables the depth information to be aligned and projected into the image plane.

Geometric representation and fusion strategy

A suitable selection of the geometric representation of the depth is a basis of fusion designs.

An exemplary geometric representation (“HHA”) encodes the depth in three channels: horizontal disparity, height above the ground and the angle that the local surface normal of the pixel forms with the direction of gravity. The estimation of the direction of gravity, the surface normal and the ground are the main components for a high quality.

Other exemplary approaches to geometric representation work entirely in the 3D area using voxel representation, which has strong potential in 3D object recognition.

Using these representations, deep learning by incorporating the 3D Cartesian position XYZ can supplement the image information to produce robust accuracy. Learning the Cartesian coordinate in a pixel through channel-wise concatenation with the image tensor enables the network to learn all or part of the translation invariance.

The embodiments of the present disclosure follow this principle and concatenate the depth as 3D Cartesian coordinates with the RGB image. This provides a fusion architecture as exemplified in 2 is shown.

In the 2 The architecture shown as an example (“deepRGBXYZ architecture”) consists of 5 layers. Each layer applies 4 folds with kernels (K) and dilatation rates (d). The resulting feature map is dense with 128 channels.

Network architecture

CNN-based solutions result in higher accuracy compared to handcrafted systems.

Many of the networks use multiples of maximal pooling and step convolution to represent the spatial coherence of the closest pixels in the resulting feature maps. They work efficiently for image classification tasks, but they significantly reduce spatial resolution and produce sparse feature maps.

Some architectures follow the resulting feature maps through bilinear interpolation to restore full resolution of the feature responses, and others provide deconvolutionary layers.

Using dilated or atrous folds is a better choice for maintaining full resolution of feature responses. They differ from the standard architectures in the changing rates of expansion of the convolution cores. This principle shows potential for semantic segmentation as well as for learning descriptors. Both approaches are stacked layers with increasing dilation rates.

This structure is used in accordance with some embodiments of the present disclosure to develop a fusion in order to improve the distinguishability of the feature maps as much as possible and to support the image regions that lack many details in their receptive fields with depth information. At the same time, the receptive field for the 3D information displayed as XYZ is enlarged, and the context information of the 2D patches is supported with 3D hints. Due to the early fusion, the 2D and 3D information share the same receptive field.

In an exemplary embodiment, 5 layers can be stacked, each employing 4 parallel folds with 5 × 5 cores and at 1, 2, 3, and 4 dilation rates. The size of the receptive field is 81 pixels, as in 2 shown.

Training details and loss function

A triplet-based network can be used in training the neural network, as shown in FIG 3 is shown. The core of this approach encodes the similar feature cards to be closer than the dissimilar ones. For this purpose the deepRGBXYZ training network accepts three parallel patches; Reference patches, positive patches, and negative patches (with common weights). The images are sampled in patches in such a way that the reference patches and positive patches are very similar, while the negative patch is viewed from a large distance from the reference patch.

Thus, the 3D information is used as input for the training. Datasets are selected that provide depth data and an Optical Flow Ground Truth to convert the images into the patches needed. From a second perspective, the Optical Flow Ground Truth thus facilitates the sampling of the positive patch, which in the first instance is strongly correlated with the reference patch. The negative patch is also obtained from the second view, but with a changed shift, which can be semi-correlated with the positive patch.

A loss function (e.g. Thresholded Hinge Embedding Loss Function) can be used to train the above-mentioned patches. The loss function tries to minimize the L2 distance between the reference patch and the positive patch and to increase the L2 distance between the reference patch and the negative patch, taking into account a minimum and maximum threshold value.

4th shows a vehicle 10 with a driver assistance system 400 for automated driving according to the embodiments of the present disclosure.

With automated driving, the vehicle is guided longitudinally and laterally 10 automatically. The driver assistance system 400 So takes over the vehicle guidance. The driver assistance system controls this 400 the drive 20th , The gear 22nd , the service brake 24 and the steering 26th via intermediate units, not shown.

To plan and implement automated driving, information about the surroundings is provided by an environmental sensor system that monitors the surroundings of the vehicle from the driver assistance system 400 accepted. In particular, the vehicle can have at least one environment sensor 12th which is set up to record environmental data indicating the vehicle surroundings. The at least one environmental sensor 12th can for example comprise one or more LiDAR systems, one or more radar systems and / or one or more cameras.

In particular, the vehicle 10 a camera and a LiDAR system, the data of which is used to determine the feature descriptor in accordance with embodiments of the present disclosure. In particular, the feature descriptors can estimate the dense 3D motion field of a scene that is moving in whole or in part with respect to the camera, for example. Reliable control of the automated driving function can thus be made possible.

According to the invention, a two-dimensional image (e.g. an RGB image) and depth information are combined in order to determine a (e.g. CCN-based) feature descriptor. In this way, in particular, improved accuracy in the case of pixel-by-pixel matching for the flow estimation can be made possible.

Although the invention has been illustrated and explained in more detail by preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that there is a multitude of possible variations. It is also clear that embodiments cited by way of example really only represent examples that are not to be interpreted in any way as a limitation, for example, of the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to specifically implement the exemplary embodiments, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without the To leave the scope of protection that is defined by the claims and their legal equivalents, such as further explanations in the description.

Claims (10)

A method (100) for generating a robust feature descriptor comprising: Capturing (110) two-dimensional image data by a first sensor system; Acquiring (120) depth measurement data by the first sensor system or a second sensor system; and Combining (130) the two-dimensional image data and the depth measurement data to generate a feature descriptor. The method (100) according to Claim 1 wherein the method (100) uses a neural network, and in particular a convolutional neural network, CNN. The method (100) according to Claim 2 , wherein the two-dimensional image data and the depth measurement data share the same receptive field. The method (100) according to Claim 2 or 3 , further comprising: training the neural network using a triplet-based network, wherein similar feature cards are encoded so that they are closer than dissimilar feature cards. The method (100) according to one of the Claims 1 until 4th , wherein combining (130) the two-dimensional image data and the depth measurement data comprises: concatenation of the depth as Cartesian 3D coordinates with an RGB image. A driver assistance method for a vehicle, comprising: controlling a driving function for automated driving using the method (100) according to one of the Claims 1 until 5 generated robust feature descriptor. Storage medium, comprising a software program which is set up to be executed on one or more processors, and to thereby use the method according to one of the Claims 1 until 6th to execute. A system for generating a robust feature descriptor, comprising a processor unit which is arranged to: receive two-dimensional image data captured by a first sensor system; Receive depth measurement data acquired by the first sensor system or a second sensor system; and combine the two-dimensional image data and the depth measurement data to generate a feature descriptor. Vehicle, in particular motor vehicle, comprising the system according to Claim 8 . The vehicle after Claim 9 , further comprising the first sensor system and the second sensor system.

Images