EP4028935A1

EP4028935A1 - Method for determining an environmental model of a scene

Info

Publication number: EP4028935A1
Application number: EP20746149.2A
Authority: EP
Inventors: Linkun XU; Bingtao Gao; Guangfu WANG; Christian Thiel
Original assignee: Continental Automotive Technologies GmbH
Current assignee: Aumovio Germany GmbH
Priority date: 2019-07-31
Filing date: 2020-07-22
Publication date: 2022-07-20
Also published as: WO2021018690A1

Abstract

A method for determining an environmental model of a scene, comprises a step of capturing a plurality of frames of the scene by a camera, and a step of evaluating the images by a simultaneous localization and mapping process (SLAM). The simultaneous localization and mapping process comprises at least the following steps of: A portion of feature points is matched between the frames. The portion of the feature points is selected in dependence on a classification of the feature points. A 3D model of the scene is generated based on the matched feature points, the 3D model being construed as a 3D point cloud.

Description

Method for determining an environmental model of a scene

Field of the invention

The embodiments relate to a method for determining an environmental model of a scene by using a simultaneous localization and mapping algorithm.

Description of the related art

Advanced driver systems and autonomously driving cars require high precision maps of roads and other areas on which vehicles can drive. Determining a vehi cle's position on a road with the high accuracy needed for self-driving cars can- not be achieved by conventional navigation systems, such as satellite navigation systems, for example GPS, Galileo or GLONASS, or other known position tech niques such as triangulation and the like. However, in particular when a self driving vehicle moves on a road with multiple lanes, it is desired to exactly de termine the position of the vehicle on one of the lanes. A popular method for positioning that is used today is known as simultaneous localization and mapping (SLAM). SLAM is one of the techniques to calculate vis ual camera pose and movement or camera trajectory, and build a SD point cloud of a scene, from a continuous image sequence. SLAM generates and uses feature points. Feature points can be salient points of salient regions in a two- dimensional image generated by a 2D-sensor, for example a camera, or in a two- dimensional representation of an environment generated by a scanning sensor.

SLAM usually tries to find feature points from one image and calculate a de scriptor of the feature points to describe a pattern around each feature point. Feature point descriptors are mathematical descriptions of the characteristics of a region in an image. They include information about the rotational and transla- tional behavior of the image region that includes the feature point. They are 2D points but include 3D information.

SLAM tries to match the feature points into pairs between image frames. Then, the camera pose and a 3D point cloud can be calculated from the feature point pairs using multiple view geometry equations. SLAM algorithms usually suffer from some drawbacks.

As a first problem, SLAM algorithms assume that the scene is static, and the camera that captures an image of the scene is the only device moving in the sce ne. SLAM algorithms only use descriptors of the feature points to match feature point pairs and, therefore, are often very sensitive to moving objects. This can cause large errors in calculated SLAM trajectories.

As a second problem, SLAM algorithms often determine false matching pairs of feature points. SLAM algorithms usually only use the descriptors of the feature points to match the feature points between frames. In theory only eight pairs of feature points are needed to calculate the pose of the camera. However, this cannot be done in reality, since there will be false matches. Although as new de scriptors are invented to increase the correct matching rate, there will be false matching pairs which also cause large errors in the calculated SLAM trajectories.

It is desired to provide a method for determining an environmental model of a scene with high accuracy so that a generation of a highly accurate 3D point cloud of the scene is possible.

Summary of the invention

A method for determining an environmental model of a scene which allows to generate the model with high accuracy is specified in claim 1. According to a method for determining an environmental model of a scene, a plurality of images or frames of the scene is captured by a camera. The captured images are evaluated by a simultaneous localization and mapping (SLAM) pro cess. The simultaneous localization and mapping process comprises at least a step of matching a portion of feature points between the frames, wherein the portion of the feature points is selected in dependence on a classification of the feature points. The simultaneous localization and mapping process further com prises a step of generating a 3D model of the scene, wherein the 3D model is construed as a 3D point cloud.

In an embodiment, the simultaneous localization and mapping process of the proposed method may include a step of extracting the respective feature points from the plurality of frames. The portion of the feature points may be selected from the extracted feature points in the simultaneous localization and mapping process such that feature points located on static objects in the scene are used for the matching step between the frames.

In another embodiment, a semantic segmentation of at least one of the frames of the scene is performed before evaluating the images by the simultaneous lo calization and mapping process. The semantic segmentation may provides classi fication information of at least a portion of the pixels of the at least one of the frames. The classification information specifies if a pixel of the at least one of the frames belongs to a predefined object included in the scene, for example if a pixel belongs to a vehicle, or a road surface or a pole, etc.

An embodiment relates to a vehicle or a stationary computer system including a microprocessor or microcontroller to perform the method described herein.

Additional features and advantages are set forth in the detailed description that follows. It is to be understood that both the foregoing general description and the following detailed description are merely exemplary and are intended to provide an overview or framework for understanding the nature and character of the claims. Solutions of the problem are described in the independent claims.

The dependent claims relate to further improvements of the invention.

Description of Drawings

In the following the invention will be described by way of example, without limi tation of the general inventive concept, on examples of embodiment with refer ence to the drawings.

Figure 1 shows a flowchart illustrating method steps of a method for determining an environmental model of a scene.

In figure 1 a first embodiment is shown. The method uses a simultaneous locali zation and mapping process for determining an environmental model of a scene. A plurality of frames/images of the scene is captured by a camera. The captured images are evaluated by using a simultaneous localization and mapping (SLAM) process. The simultaneous localization and mapping process may include a step of matching a portion of feature points between the frames, wherein the portion of the feature points is selected in dependence on a classification/ classes of the feature points. A 3D model of the scene is generated based on the matched fea ture points. The 3D model is construed as a 3D point cloud.

After determining the 3D point cloud, further processing steps can be carried out. For example, a vector model can be generated from the 3D point cloud.

The method for determining an environmental model of a scene is explained in the following in detail with reference to the figure showing a flowchart of steps SI to S5 of the proposed method. The method may be performed by a control unit which may be embodied as a processor or a controller. This processor or controller may be part of a vehicle or of a stationary computer system. In a first step SI, after having captured the plurality of frames/images of the sce ne by a camera, a semantic segmentation of at least one of the frames of the scene, for example an input frame, is performed, before evaluating the images by the simultaneous localization and mapping process.

Further, deep neural networks may be used for image semantic segmentation. Method step SI is configured as a step of deep learning by using a deep neural network to proceed with semantic segmentation before executing the simulta neous localization and mapping process. The input is an image frame, and the output is a semantic segmentation of that image frame. Other techniques, like Pattern Matching, can also be used for classification of the feature points.

The semantic segmentation will give classification information or several classifi cations with top confidences of each pixel of an input image frame. In particular, the semantic segmentation provides classification information of each pixel of the at least one of the frames, i.e. the input frame. The classification information specifies if a respective one of the pixels of the at least one of the frames belongs to a predefined object included in the scene, such as a vehicle, or a road surface or a pole, etc. The classification information will then be passed into the SLAM module/algorithm comprising method steps S2 to S5.

In method step S2 of the simultaneous localization and mapping process, respec tive feature points are extracted from the plurality of frames. To extract the fea ture points of the frames, state of the art feature point extraction algorithms can be used.

After the feature point extraction step S2 in the SLAM processing, a feature point selection step SB is performed. According to a possible embodiment of the method, the portion of the feature points of a frame to be used for matching is selected from the extracted feature points of step S2 in the simultaneous locali zation and mapping process such that feature points having the same confidence classification are used for the matching step between the frames.

In particular, the portion of the feature points to be considered for matching may be selected from the extracted feature points in the simultaneous localization and mapping process such that feature points located on static objects in the scene are used for the matching step between the frames. That means that the classification information obtained from the step SI of semantic segmentation may be utilized to remove those feature points being classified as belonging to a possible moving object, for example a vehicle in which the camera is mounted, other vehicles, pedestrians, bicycles, motorbikes, etc.. In conclusion, after having performed the feature point selection step SB, only feature points located on static objects are selected to be matched in a subsequent step S4 of the simulta neous localization and mapping process. Step S4 of the simultaneous localization and mapping process may include the matching of a portion of the feature points between the frames. The portion of the feature points used for matching that portion which has been selected in dependence on the classification of the feature points provided in step SI by the semantic segmentation. Feature points having the same confidence classification may be used for the matching step between the frames. In an preferred embod iment, feature points located on static objects in the scene are used for the matching step between the frames.

In an embodiment, the portion of the feature points is matched between the frames in the SLAM process by using respective descriptors of the portion of the feature points. Instead of performing a matching of feature points using only descriptors of the feature points as in a conventional SLAM process, the clas- ses/classifications of the feature points determined in step SI of the semantic segmentation may also considered in the matching step S4.

Since feature points with the same high confidence classifications are used to be matched between the frames, feature points on an object in a previous frame may only be matched with the feature points on the same type of object in the next frames.

In an embodiment, the portion of the feature points may be matched between the frames in the simultaneous localization and mapping process by using re spective descriptors of the portion of the feature points after the step of seman- tic segmentation, wherein the descriptors of the portion of the feature points are calculated in dependence on the classification information. The calculation of the descriptors of feature points is a very complex and computationally intensive process. If, on the other hand, the classification information obtained from the semantic segmentation is taken into account, only descriptors of feature points that lie on objects of particular interest may be calculated.

In a step S5 of the simultaneous localization and mapping process, a 3D model of the scene may be generated based on the matched feature points obtained from step S4. The 3D model may be construed as a 3D point cloud. According to a pos sible embodiment of the method for determining an environmental model of a scene, a camera pose of the camera may be determined in step S5 by evaluating the matched feature points. Furthermore, a 3D position of the matched feature points is determined in the scene.

In addition to determining the camera pose and the calculation of the 3D posi tion of the feature points, an optimization may be performed in step S5. Step S5 basically stays the same with state-of-the-art feature point extraction algorithms. When compared to a conventional SLAM process the steps SI, SB and S4 are added to determine the environmental model of the scene to improve the accu racy of the calculated model of the scene. The proposed method for determining an environmental model of a scene combines the advantages of the semantic segmentation technology using deep neural networks and a simultaneous locali zation and mapping process.

By adding step SI and step S3 in a traditional SLAM processing, only feature points on static objects are used for the feature point matching of step S4. This matches the assumption of the simultaneous localization and mapping process that vehicle trajectory is closer to the ground truth, and a more accurate 3D point cloud can be generated.

Moreover, by adding method steps SI and S4 in the traditional SLAM processing, additional conditions are used in the matching step to get more accurate feature point matching between the frames so that the false matching rate is decreased and the vehicle trajectory shape is further improved compared with ground truth. A more accurate 3D point cloud can be generated.

In summary, the method steps SI, S3 and S4 of the proposed method for deter mining an environmental model of a scene improves the vehicle trajectory posi tion and 3D point cloud accuracy. The proposed method has still some other benefits. With less objects to be taken into account in the image for feature point generation and/or matching, CPU resources can be saved and the program by which the method is implemented can run faster. Since the method allows to filter-out 3D points that belong to non-static objects, such 3D points will no longer be reported to the server side. The method further allows to provide a large number of 3D points on road sur face, poles, barriers, traffic signs, and less on moving objects, for example on trees. In conclusion, the 3D points can be more robust and invariable over the season.

According to a possible embodiment of the method for determining the envi ronmental model of a scene, the semantic segmentation information obtained in step SI can be used in a direct method SLAM process. In this case, the portions of the feature points are matched between the frames in the simultaneous local ization and mapping process by using a respective luminescence of pixels of the frames instead of using respective descriptors of the portion of the feature points. The proposed method for determining an environmental model of a scene may be extended by using the generated point clouds to generate 3D objects. In par ticular, semantic segmentation results of method step SI can help to generate 3D objects with semantic information.

Claims

1. A method for determining an environmental model of a scene, comprising:

- capturing a plurality of frames of the scene by a camera,

- evaluating the images by a simultaneous localization and mapping process, the simultaneous localization and mapping process com prises at least the following steps of:

- matching a portion of feature points between the frames, the por tion of the feature points being selected in dependence on a classi fication of the feature points; and

- generating a 3D model of the scene based on the matched feature points, the 3D model being construed as a 3D point cloud.

2. The method of claim 1,

wherein the portion of the feature points are matched between the frames in the simultaneous localization and mapping process by using respective descriptors of the portion of the feature points.

3. The method of claim 1 or 2,

wherein the portion of the feature points are matched between the frames in the simultaneous localization and mapping process by using a respective luminance of pixels of the frames.

4. The method of any of the claims 1 to 3,

wherein the simultaneous localization and mapping process comprises a step of extracting the respective feature points from the plurality of frames.

5. The method of claim 4,

wherein the portion of the features points is selected from the extracted feature points in the simultaneous localization and mapping process such that feature points having the same confidence classification are used for the matching step between the frames.

6. The method of claim 4 or 5,

wherein the portion of the features points is selected from the extracted feature points in the simultaneous localization and mapping process such that feature points located on static objects in the scene are used for the matching step between the frames.

7. The method of any of the claims 1 to 6, comprising:

- performing a semantic segmentation of at least one of the frames of the scene before evaluating the images by the simultaneous lo calization and mapping process.

- providing classification information of at least a portion of the pixels of the at least one of the frames by the semantic segmentation.

8. The method of claim 7,

wherein the classification information specifies if a pixel of the at least one of the frames belongs to a predefined object included in the scene.

9. The method of claim 7 or 8,

wherein the portion of the feature points are matched between the frames in the simultaneous localization and mapping process by using respective descriptors of the portion of the feature points after the step of semantic segmentation, wherein the descriptors of the portion of the feature points are calculated in dependence on the classification information.

10. The method of any of the claims 1 to 9, comprising:

- determining a camera pose of the camera by evaluating the

matched feature points, and

- determining a 3D position of the matched feature points in the sce ne.

11. A vehicle including a microprocessor or microcontroller being configured to perform the method of any of the claims 1 to 10.

12. A stationary computer system including a microprocessor or microcontrol ler being configured to perform the method of any of the claims 1 to 10.