FR3138944A1 - Device and method for estimating the traversability of terrain by a mobile system - Google Patents

Abstract

The invention relates to a terrain mapping device intended to be on board a mobile system, comprising a traversability estimation unit configured to: determine (E1) a depth map of the terrain; determine (E3) a height map of the terrain from the depth map and image data of the terrain acquired by a camera on board the mobile system; determine (E4) a first mask of traversability of the terrain by the mobile system by means of thresholding of the height map; determine (E5) a second terrain traversability mask by the mobile system by means of a semantic segmentation of the terrain image data acquired by the camera on board the mobile system; determine (E6) a height map traversability of the terrain by the mobile system (CT) by merging the first and second traversability mask. Figure for abstract: Figure 1

Description

Device and method for estimating the traversability of terrain by a mobile system

The field of the invention is that of aiding the navigation of a mobile system of the robot or autonomous vehicle type. The invention relates more particularly to the estimation of the traversability of terrain by the mobile system, that is to say the identification of areas of the terrain over which the mobile system is able to move.

In the field of navigation of mobile systems, we know methods which aim to find the presence of a road in images acquired by a camera on board the mobile system. These methods use visual clues such as vanishing points, textures or even relief to delineate the contours of a road on an image, or pose the problem directly as a road segmentation problem in the image. However, these methods are not interested in paths in the broadest sense of the term, which may in particular be off-road paths that are not necessarily paved or correctly demarcated, and even less in the more general theme of traversability corresponding to identification. in the images acquired of areas of the ground over which the mobile system is able to move.

There are also methods that identify the type of ground on which the vehicle is traveling. For example, document US 2019/0129435 A1 aims to predict the traversability of terrain for a vehicle by means of sensors embedded in the vehicle such as an inertial unit, a GPS as well as a lidar to identify the topography of the ground surface and generate a 3D map of the terrain. The position of the vehicle relative to this map is determined by means of a simultaneous localization and mapping algorithm (known in English as SLAM for Simultaneous Localization And Mapping ). Then the path to be followed by the vehicle is determined based on terrain characteristics extracted from sensor data, such as data from an accelerometer or a gyroscope.

The invention aims to propose a technique for estimating the traversability of terrain by a mobile system which is both efficient and simple to implement, whether or not there is a road on the terrain. (for example in the case of off-piste navigation or in an unstructured environment).

• déterminer une carte de profondeur du terrain ;
• déterminer une carte de hauteur du terrain à partir de la carte de profondeur et de données d’image du terrain acquises par une caméra embarquée sur le système mobile ;
• déterminer un premier masque de traversabilité du terrain par le système mobile au moyen d’un seuillage de la carte de hauteur ;
• déterminer un deuxième masque de traversabilité du terrain par le système mobile au moyen d’une segmentation sémantique des données d’image du terrain acquises par la caméra embarquée sur le système mobile ;
• déterminer une carte de traversabilité du terrain par le système mobile par fusion du premier et du deuxième masque de traversabilité.

To this end, the invention proposes a terrain mapping device intended to be embedded on a mobile system, comprising a traversability estimation unit configured to:

• determine a depth map of the terrain;
• determining a height map of the terrain from the depth map and image data of the terrain acquired by a camera on board the mobile system;
• determine a first terrain traversability mask by the mobile system by means of thresholding of the height map;
• determine a second terrain traversability mask by the mobile system by means of a semantic segmentation of the terrain image data acquired by the camera on board the mobile system;
• determine a terrain traversability map by the mobile system by merging the first and second traversability mask.

• l’unité d’estimation de traversabilité est configurée pour déterminer la carte de profondeur du terrain à partir des données d’image du terrain acquises par la caméra ;
• pour déterminer la carte de profondeur l’unité d’estimation de traversabilité exploite un modèle d’apprentissage profond ;
• l’unité d’estimation de traversabilité comprend un réseau neuronal pré-entrainé pour mettre en œuvre le modèle d’apprentissage profond ;
• le réseau neuronal est également pré-entrainé pour réaliser la segmentation sémantique des données d’image du terrain ;
• le réseau neuronal comprend un encodeur, un premier décodeur dédié à l’inférence de profondeur et un deuxième décodeur dédié à la segmentation sémantique ;
• le seuillage de la carte de hauteur est réalisée de manière à ce que le premier masque de traversabilité conserve, lorsqu’il est appliqué aux données d’image du terrain, des points du terrain présentant une hauteur comprise entre un premier seuil et un deuxième seuil supérieur au premier seuil ;
• l’unité d’estimation de traversabilité est en outre configurée pour déterminer un troisième masque de traversabilité qui conserve, lorsqu’il est appliqué aux données d’image du terrain, des points du terrain présentant une hauteur inférieure au premier seuil et dans lequel déterminer la carte de traversabilité comprend en outre l’addition du troisième masque de traversabilité à la fusion du premier et du deuxième masque de traversabilité.

Some preferred but non-limiting aspects of this device are as follows:

• the traversability estimation unit is configured to determine the terrain depth map from the terrain image data acquired by the camera;
• to determine the depth map, the traversability estimation unit uses a deep learning model;
• the traversability estimation unit includes a neural network pre-trained to implement the deep learning model;
• the neural network is also pre-trained to perform semantic segmentation of the terrain image data;
• the neural network comprises an encoder, a first decoder dedicated to depth inference and a second decoder dedicated to semantic segmentation;
• the thresholding of the height map is carried out in such a way that the first traversability mask preserves, when applied to the terrain image data, points on the terrain having a height between a first threshold and a second threshold greater than the first threshold;
• the traversability estimation unit is further configured to determine a third traversability mask which retains, when applied to the terrain image data, terrain points having a height less than the first threshold and in which to determine the traversability map further comprises the addition of the third traversability mask to the merger of the first and the second traversability mask.

Other aspects, aims, advantages and characteristics of the invention will appear better on reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the appended drawings. on which ones :

- there is a diagram illustrating different steps implemented during a method according to the invention for estimating the traversability of terrain by a mobile system;

- there is an image acquired by a camera;

- there is a depth map of the image of the ;

- there is a height map determined from the depth map of the and the image of the ;

- there illustrates a first traversability mask;

- there illustrates a semantic segmentation of the image of the ;

- there is another representation of the first traversability mask;

- there illustrates the result of merging the first and second traversability mask.

DETAILED DESCRIPTION OF PARTICULAR EMBODIMENTS

The invention relates to a terrain mapping device intended to be embedded on a mobile system, for example an off-road type land mobile system such as a robot, a drone or an autonomous vehicle. As will be detailed below, this device includes a traversability estimation unit configured to achieve the fusion of a geometric navigation solution taking into account the height of obstacles encountered on the ground and a semantic navigation solution identifying the areas traversable by the mobile system in image data of the terrain acquired by a camera on board the mobile system, typically a monocular camera. The traversability estimation unit delivers a traversability map, for example a binary map in which each point of the terrain imaged by the camera is identified as being traversable or not by the mobile system or even a map in which a probability of traversability is associated with each point on the ground.

The images acquired by the camera are typically RGB images of the terrain. The device is then functional in visible light. In a variant embodiment, the functionality of the device is extended to nighttime operation by using another wavelength range (typically infrared).

In reference to the , the traversability estimation unit is configured to determine a depth map of the terrain during a step E1.

In a preferred embodiment, the determination of the depth map of the terrain during this step E1 is carried out directly from the image data of the terrain It acquired by the camera. More particularly, this determination can be obtained by means of a supervised learning model, such as a model based on a deep neural network.

Figures 2 to 8 are given as an illustrative example of the invention. There thus represents an image of the public NYU Depth v2 dataset while the represents the corresponding depth map.

In a variant embodiment, the depth map can be determined by the traversability estimation unit by means of a direct measurement by a dedicated emissive sensor (such as a LiDAR or a 3D scanner with structured light). In another variant, this determination is still carried out passively but using not one but several cameras, for example cameras of a stereovision system. It will be noted that the use of a single monocular camera has the advantages of low consumption for an on-board system, of reduced cost compared to a stereovision system or a LiDAR type system and of allowing the development of a non-emissive solution.

The depth map thus determined makes it possible to supplement the information on the terrain obtained by the camera to thus provide knowledge of the 3D structure of the terrain. By assuming that the mobile system moves on the ground, the traversable areas of the terrain then correspond to the ground plane unlike the obstacles encountered.

One of the common principles in mobile robotics to address the navigability problem of a system is to define an occupation grid for the robot's environment. The basic idea of the occupancy grid is to represent a map of the environment as a uniformly spaced field of random variables each representing the probability of the presence of an obstacle at that location in the environment. Based on this principle, the invention considers the image acquired by the camera as a map of the environment where each pixel corresponds to a point on the grid. By knowing the 3D structure of the terrain, the invention associates with each point of the grid the height of the element present at this point to decide whether this element is an obstacle or not.

To do this, the traversability estimation unit is configured to determine during a step E3 a height map of the terrain from the depth map and the image data of the terrain. With the image data of the terrain, the depth map makes it possible to calculate the 3D coordinates of each point on the terrain, coordinates which can be reprojected along the axis corresponding to the height of these points in order to calculate the height map . The determination of the height map may in particular include an inverse projective transformation based on the projective model of the camera, for example in the form of a linear matrix operation. There illustrates the height map determined from the depth map of the and the image of the .

This determination of the height map can be carried out as follows. We consider the coordinates of a pixel in the original images showing geometric distortions. In order to determine the elevation (height) of this pixel, it is necessary to know the intrinsic parameters of the camera, namely the projection matrix (focal length and optical center) of the camera as well as the distortion parameters . These parameters are conventionally determined by camera calibration. A first step aims to correct the pixel distortion by applying the distortion model determined during calibration. For example in the case of a radial distortion model, the distortion correction is carried out by the following equation:

with represents the center of the distortion.

A second step is to determine the 3D position of each pixel in 3D space, by applying the inverse projection and using the estimated depth, as follows:

With :

the position of the 3D point in the camera frame,

the camera parameters (focal length and optical center), and

the estimated depth.

Point being determined in the camera's reference frame, it is possible to determine its height relative to the ground, because the position of the camera in the vehicle is known a priori (distance between the camera and the wheels of the vehicle).

In a possible embodiment in which the mobile system embeds an inertial unit, the traversability estimation unit can, in a step E2 prior to step E3, adapt the projective model to the pose of the camera such that estimated by the inertial unit.

As indicated previously, the height that we wish to estimate is in relation to the ground and not in the camera's reference frame. Thus, it is necessary to take into account the pose of the camera. To do this, simply take a flat ground as a reference and estimate the position of the camera each time an image is taken relative to an initial position using inertial data. This amounts to estimating the relative poses of the camera compared to an initial position using inertial data.

In particular, this adaptation makes it possible to standardize the view of the terrain in order to eliminate biases linked to the orientation of the vehicle in relation to the ground. A change of reference, from the camera reference to the imaged terrain reference, can in particular be carried out by means of the different angles provided by the inertial unit.

In a step E4, the traversability estimation unit is configured to determine a first terrain traversability mask by the mobile system by means of thresholding of the height map. This thresholding makes it possible to determine for each pixel of the height map whether the point on the ground associated with this pixel can be traversed or not by the mobile system, that is to say whether it is part of the ground or whether, on the contrary, its height indicates that it is an obstacle. This thresholding is thus carried out according to the characteristics of the mobile system according to the dimensions of the obstacles to be avoided by it.

In one possible embodiment, the first traversability mask preserves, when applied to the terrain image data, terrain points having a height less than a threshold height. There illustrates in this regard the thresholding of the height map of the to determine the first traversability mask, which preserves when applied to the terrain image data the terrain points associated with the white areas on the .

In another possible embodiment, the first traversability mask is determined so that it preserves, when applied to the terrain image data, terrain points having a height between a first threshold and a second threshold greater than the first threshold. In this other embodiment, another traversability mask, called third traversability mask in the following, can be determined which preserves, when applied to the terrain image data, the terrain points having a lower height at the first threshold.

The traversability estimation unit is also configured to, during a step E5, determine a second terrain traversability mask by the mobile system by means of a semantic segmentation of the terrain image data acquired by the on-board camera on the mobile system. There illustrates in this regard the result of a semantic segmentation of the image of the in which the areas of the terrain identified by semantic segmentation with the category “traversable by the mobile system” bear the reference Tsm. This semantic segmentation can be obtained using a supervised learning model, such as a deep neural network-based model. In one possible embodiment, this segmentation is binary, the pixels of the image being segmented into two distinct classes: traversable by the mobile system and non-traversable by the mobile system.

The geometric information represented by the first traversability mask and the semantic information represented by the second mask prove to be complementary.

Indeed, on the one hand, objects of small size or of which a part has a height included in the height range authorized for the determination of the first traversability mask are not taken into account as obstacles when only the geometric information is used. On the other hand, by coupling the geometric information with the semantic information indicating that these objects or parts of objects are not traversable, it turns out to be possible to correctly identify them as obstacles.

Furthermore, on the other hand, semantic segmentation does not alone solve the traversability estimation problem since it is only based on the appearance of the image. However, traversable zones represent a potentially complex class, particularly due to varied textures (earth, grass or even bitumen are three types of potentially traversable soil but give very different results and textures). Furthermore, uncorrelated from geometric notions, semantic segmentation can provide aberrant results such as “traversable” pixels in improbable locations whose homogeneous texture is locally similar to that of the floor (for example the section of wall between the bed and the desk identified in the zone bearing the reference ZA on the ). On the other hand, by coupling the semantic information with the geometric information indicating that these places are not traversable, it turns out to be possible to correctly identify them as obstacles.

In accordance with the invention, the traversability estimation unit is advantageously configured to, during a step E6, determine a CT traversability map by merging the first and the second traversability mask. In this way, only the pixels of the environmental grid which are both preserved by the first and the second traversability mask are ultimately considered as traversable by the mobile system. This fusion typically includes the application of a logical AND between the first and the second traversability mask and thus makes it possible to retain only the geometrically viable pixels whose semantics are proven.

The problems of small obstacles present in the range of values used for determining the first traversability mask are thus resolved, while removing aberrant pixels whose predicted semantic class is “traversable” at improbable locations in the image. . There illustrates in this regard the first traversability mask M1 while the illustrates the fusion of the first mask of the with the second mask to provide the CT traversability map. We see that the traversability estimate is refined. In particular, the legs of the chair are cut out and are not considered as traversable pixels whereas they were on the . Furthermore, no pixel whose height is too high in the image is proposed in the final estimate (see the section of wall in zone ZA of the ).

In the previously mentioned embodiment where the first traversability mask is determined so that it preserves, when applied to the terrain image data, the terrain points having a height between the first and the second threshold and where a third traversability mask is determined so that it preserves, when applied to the terrain image data, the terrain points having a height less than the first threshold, only the first mask is merged with the second mask and the third mask is then added to the result of this fusion. Such an achievement makes it possible to consider that below the first height threshold, all the pixels (of the third mask) are considered traversable while on the rest of the image the pixels considered to be traversable are those resulting from the fusion of the first mask (geometric approach) and the second mask (semantic approach).

We have seen in the above that the traversability estimation unit can exploit a deep learning model to determine the depth map.

This deep learning model can leverage a neural network that features an encoder-decoder architecture. The encoder is responsible for extracting features which are subsequently provided to the decoder to generate the depth maps. The feature extractor includes successive convolution layers followed by nonlinearities such as data normalization functions, dimension reduction functions or nonlinear reprojection functions such as, among others, the sigmoid or the rectified linear unit (convolutional network). The reduced-dimensional characteristics extracted from the image are then delivered to the decoder (it is also typically a convolutional network) which is responsible for recovering the spatial dimension of the image while calculating the characteristics necessary for decoding the characteristics, thus producing a map depth.

• Acquisition d’une base de données d’apprentissage qui comporte des images de scènes ainsi que les cartes de profondeurs qui leurs correspondent. Ces cartes de profondeur vérité-terrain sont obtenue par une méthode tierce, par exemple une carte de profondeur acquise par LIDAR, au moyen d’une lumière structurée, par stéréovision, etc.
• Calcul d’une valeur représentative de la performance du modèle d’apprentissage par comparaison de la carte de profondeur de la scène fournie en sortie du réseau de neurones et la carte de profondeur vérité-terrain.
• Les poids des connexions du réseau de neurone sont alors ajustés de sorte à réduire l’erreur de prédiction de la carte de profondeur. Par exemple, le gradient de l’erreur peut être calculé afin de déterminer une direction de variation et un déplacement dans une direction opposée au gradient est alors réalisée.

Training of the deep learning model can be carried out as follows:

• Acquisition of a learning database which includes scene images as well as the depth maps that correspond to them. These ground truth depth maps are obtained by a third-party method, for example a depth map acquired by LIDAR, by means of structured light, by stereovision, etc.
• Calculation of a value representative of the performance of the learning model by comparison of the depth map of the scene provided as output from the neural network and the ground truth depth map.
• The weights of the neural network connections are then adjusted so as to reduce the prediction error of the depth map. For example, the gradient of the error can be calculated to determine a direction of variation and a movement in a direction opposite to the gradient is then carried out.

Thus, in one possible embodiment, the traversability estimation unit may comprise a neural network pre-trained to implement the deep learning model. In a possible variant, this neural network can also be pre-trained to perform semantic segmentation of terrain image data. In particular, the neural network may include an encoder and, at the output of the encoder, a first decoder dedicated to determining the depth map and a second decoder dedicated to semantic segmentation.

The invention is not limited to the device previously described but also extends to a method during which the steps previously described are implemented by computer as well as to a computer program product comprising instructions which, when the program is executed by a computer, lead it to implement the steps of such a process.

Claims (10)

Terrain mapping device intended to be embedded on a mobile system, comprising a traversability estimation unit configured to:

• determine (E1) a depth map of the terrain;
• determine (E3) a height map of the terrain from the depth map and image data of the terrain acquired by a camera on board the mobile system;
• determine (E4) a first terrain traversability mask by the mobile system by means of thresholding of the height map;
• determine (E5) a second terrain traversability mask by the mobile system by means of a semantic segmentation of the terrain image data acquired by the camera on board the mobile system;
• determine (E6) a terrain traversability map by the mobile system (CT) by merging the first and second traversability mask.

Device according to claim 1, wherein the traversability estimation unit is configured to determine the terrain depth map from the terrain image data acquired by the camera. Device according to claim 2, in which to determine the depth map the traversability estimation unit uses a deep learning model. Device according to claim 3, wherein the traversability estimation unit comprises a neural network pre-trained to implement the deep learning model. Device according to claim 4, wherein the neural network is also pre-trained to perform semantic segmentation of the terrain image data. Device according to claim 5, in which the neural network comprises an encoder, a first decoder dedicated to depth inference and a second decoder dedicated to semantic segmentation. Device according to one of claims 1 to 6, in which the thresholding of the height map is carried out in such a way that the first traversability mask preserves, when applied to the terrain image data, points of the land having a height between a first threshold and a second threshold greater than the first threshold. A device according to claim 7, wherein the traversability estimation unit is further configured to determine a third traversability mask which preserves, when applied to the terrain image data, terrain points having a height lower than the first threshold and in which determining the traversability map further comprises adding the third traversability mask to the merger of the first and the second traversability mask. Computer-implemented method for estimating the crossability of terrain by a mobile system, comprising the steps of:

• determine a depth map of the terrain;
• determining a terrain height map from the depth map and terrain image data acquired by a camera on the mobile system;
• determine a first terrain traversability mask by the mobile system by means of thresholding of the height map:
• determine a second terrain traversability mask by the mobile system by means of a semantic segmentation of terrain image data acquired by the camera on board the mobile system;
• determine a terrain traversability map by the mobile system by merging the first and second traversability mask.

Computer program product comprising instructions which, when the program is executed by a computer, lead it to implement the steps of the method according to claim 9.