EP4115318A1

EP4115318A1 - System and method for predicting a map from an image

Info

Publication number: EP4115318A1
Application number: EP20711531.2A
Authority: EP
Inventors: Thomas RODDICK; Roberto Cipolla; Daniel OLMEDA REINO
Original assignee: Toyota Motor Europe NV SA; Cambridge Enterprise Ltd
Current assignee: Cambridge Enterprise Ltd; Toyota Motor Corp
Priority date: 2020-03-05
Filing date: 2020-03-05
Publication date: 2023-01-11
Also published as: WO2021175434A1

Abstract

The invention relates to system for predicting a map from an image, comprising a first module comprising a trained neural network and configured to input the image to the neural network, wherein the neural network is configured to predict an occupancy grid map, which allows for each grid cell of the map to simultaneously encode the presence of multiple object classes

Description

SYSTEM AND METHOD FOR PREDICTING A MAP FROM AN IMAGE

FIELD OF THE DISCLOSURE

[0001] The present disclosure is related to the field of image processing, in particular to a system and method for predicting a map from an image.

BACKGROUND OF THE DISCLOSURE

[0002] Processing images may be used to detect various objects visible on the image. Such processing is therefore particularly useful for autonomous vehicles (partially or fully self-driving) and for other types of automated systems, because these systems have to be able to understand their surroundings on the basis of images acquired by cameras.

[0003] Autonomous vehicles commonly rely on highly detailed birds-eye- view maps of their environment, which capture both static elements of the scene such as road layout and traffic sign locations; as well as dynamic elements such as other cars and pedestrians. Generating these map representations on the fly is a complex multi-stage process which incorporates many important vision-based elements, including ground plane estimation, road segmentation and 3D object detection.

[0004] In particular, autonomous vehicles and other robotic platforms require a rich, succinct and detailed representation of their environment which captures both the geometry and layout of the static world as well as the pose and dimensions of other dynamic agents. Such representations often provide the foundation for all decision making, including path planning, collision avoidance and navigation. Rather than capturing the full 3D world in its entirety, one popular solution is to represent the world in the form of a birds- eye-view (BEV) map, which provide a compact way to capture the spatial configuration of the scene. Such maps are convenient in that they are simple to visualise and process, exploiting the fact that in many scenarios the essential information for navigation is largely confined to the ground plane

[0005] Construction of these birds-eye-view maps is at present a complex multistage processing pipeline, involving the composition of multiple fundamental machine vision tasks: structure from motion, ground plane estimation, road segmentation, lane detection, 3D object detection, and many more. Intuitively, however, all these tasks are related: knowing the layout of the road ought to inform one about where in the image one should look for cars; and similarly a car emerging from behind a building may indicate the presence of a hidden side road beyond. There therefore seems to be a clear imperative towards discarding this complicated multistage pipeline and replacing it with a simple end-to-end approach which is able to reason holistically about the world and predict the desired map representation directly from sensor observations. In this work one may focus on the particularly challenging scenario of BEV map estimation from monocular surround view images alone. This is important given the high cost of existing range sensors and the higher resolution of cameras which allows them to perceive small objects further into the distance.

[0006] There have been several works in the field of map representations for autonomous driving. In particular, high definition birds-eye-view maps have been shown to be an extremely powerful representation across a range of different driving tasks. In 3D object detection, e.g. Yang et al. use ground height prior information from maps to improve the quality of input LiDAR point clouds, cf.:

Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In Conference on Robot Learning, pages 146-155, 2018.

[0007] Ma et al. correlate visual observations with sparse HD map features to perform highly accurate localization, cf.:

Wei-Chiu Ma, Ignacio Tartavull, loan Andrei Barsan, Shenlong Wang, Min Bai, Gellert Mattyus, Namdar Homayounfar, Shrinidhi Kowshika Lakshmikanth, Andrei Pokrovsky, and Raquel Urtasun. Exploiting sparse semantic hd maps for self-driving vehicle localization. arXiv preprint arXiv: 1908.03274, 2019.

[0008] Birds-eye-view maps are particularly valuable in the context of prediction and planning given their metric nature: Djurik et al. and Casas et al. render the local environment as a rasterised top-view map representation, incorporating road geometry, lane direction, and traffic agents, and use this representation to predict future vehicle trajectories, cf.:

Nemanja Djuric, Vladan Radosavljevic, Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung-Han Lin, and Jeff Schneider. Motion prediction of traffic actors for autonomous driving using deep convolutional networks. arXiv preprint arXiv: 1808.05819, 2018, and Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. In Conference on Robot Learning, pages 947- 956, 2018.

[0009] A similar representation is used by Bansal et al. as input to their imitation learning pipeline, allowing an autonomous agent to drive itself by recursively predicting its future state, cf.:

Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv: 1812.03079, 2018.

[0010] Hecker et al. augment their camera-based end-to-end driving model with a rendered map view from a commercial GPS route planner and show that this significantly improves driving performance, cf.:

Simon Hecker, Dengxin Dai, and Luc Van Gool. End-to-end learning of driving models with surround-view cameras and route planners. In Proceedings of the European Conference on Computer Vision (ECCV), pages 435-453, 2018.

[0011] Furthermore, there have been several works in the field of top-down representations from images. In particular, a number of prior works have tackled the difficult problem of predicting birds-eye-view representations directly from monocular images. A common approach is to use inverse perspective mapping (IPM) to map image features onto the ground plane via a homography, cf. e.g.:

Syed Ammar Abbas and Andrew Zisserman. A geometric approach to obtain a bird's eye view from an image. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019, and Chien-Chuan Lin and Ming-Shi Wang. A vision based top-view transformation model for a vehicle parking assistant. Sensors, 12(4): 4431-4446, 2012.

[0012] Other works focus on the birds-eye-view object detection task, learning a mapping from bounding box detections in the image space to bounding boxes in the birds-eye-view space, cf. e.g.:

Andrea Palazzi, Guido Borghi, Davide Abati, Simone Calderara, and Rita Cucchiara. Learning to map vehicles into bird's eye view. In International Conference on Image Analysis and Processing, pages 233-243. Springer, 2017, and Dequan Wang, Coline Devin, Qi-Zhi Cai, Philipp Kr^' ahenb^' uhl, and Trevor Darrell. Monocular plan view networks for autonomous driving. arXiv preprint arXiv: 1905.06937, 2019.

[0013] Relatively few works have tackled directly the problem of predicting a birds-eye-view semantic occupancy grid from monocular images. Some use the IPM approach mentioned above to map a semantic segmentation of the image plane into the birds-eye-view space, an approach which works well for estimating local road layout but which fails for objects such as cars and pedestrians which lie above the ground plane, cf. e.g.:

Liuyuan Deng, Ming Yang, Hao Li, Tianyi Li, Bing Hu, and Chunxiang Wang. Restricted deformable convolution-based road scene semantic segmentation using surround view cameras. IEEE Transactions on Intelligent Transportation Systems, 2019, and

Timo Samann, Karl Amende, Stefan Milz, Christian Witt, Martin Simon, and Johannes Petzold. Efficient semantic segmentation for visual bird's-eye view interpretation. In International Conference on Intelligent Autonomous Systems, pages 679-688. Springer, 2018.

[0014] The VED method of Lu et al. uses a variational encoder-decoder network to predict a semantic occupancy grid directly from an image, cf.: Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman. Monocular semantic occupancy grid mapping with convolutional variational encoder-decoder networks. IEEE Robotics and Automation Letters, 4(2): 445-452, 2019.

[0015] The use of a fully-connected bottleneck layer in the network however means that much of the spatial context in the network is lost, leading to an output which is fairly coarse and is unable to capture small objects such as pedestrians. Panet al. adopt a similar approach, predicting a birds-eye-view semantic segmentation from a stack of surround view images, via a fully- connected view-transformer module, cf.:

Bowen Pan, Jiankai Sun, Alex Andonian, Aude Oliva, and Bolei Zhou. Cross-view semantic segmentation for sensing surroundings. arXiv preprint arXiv: 1906.03560, 2019.

[0016] Schulter et al. propose to use an in-painting CNN to infer the semantic labels and depth of the scene behind foreground objects, and generate a birds-eye-view by projecting the resulting semantic point cloud onto the ground plane, cf.:

Samuel Schulter, Menghua Zhai, Nathan Jacobs, and Manmohan Chandraker. Learning to look around objects for top-view representations of outdoor scenes. In Proceedings of the European Conference on Computer Vision (ECCV), pages 787-802, 2018.

[0017] However, given the lack of ground truth data, many of the above methods rely on weak supervision from stereo (cf. Lu et al. as cited above), weakly-aligned map labels (cf. Schulter et al. as cited above) or synth-to-real domain transfer (cf. Schulter et al. and Pan et al. as cited above).

SUMMARY OF THE DISCLOSURE

[0018] Currently, it remains desirable to provide a system and a method for predicting a map from an image (e.g. birds-eye-view maps of the environment of the system) which overcomes one or more deficiencies of the prior art. In particular, it remains desirable to thereby provide a simple, unified approach for estimating these map representations directly from raw sensor data using a single end-to-end deep learning architecture. More in particular, it is desirable to provide a system and a method which allow generating such maps from monocular images alone, where no direct depth information is available.

[0019] Therefore, according to the embodiments of the present disclosure, a system for predicting a map from an image, comprising a first module comprising a trained neural network and configured to input the image to the neural network, wherein the neural network is configured to predict an occupancy grid map, which allows for each grid cell of the map to simultaneously encode the presence of multiple object classes.

[0020] In other words, the occupancy grid map is configured to simultaneously encode for each grid cell of the map the presence of multiple object classes.

[0021] By providing such a system, it becomes possible to to predict semantic occupancy grid maps on real data using a directly supervised approach.

[0022] The map predicted from the image may be a semantic map. More in particular, the occupancy grid map is a semantic Bayesian occupancy grid map. [0023] Desirably the neural network has a probabilistic occupancy grid framework. Occupancy grid maps are widely used in robotics, and allow one to trivially incorporate information over multiple sensors and timesteps. They are also conveniently processed by convolutional neural networks, allowing to take advantage of powerful new developments from the deep learning technology. In the present disclosure the traditional definition of occupancy grids, which usually only consider the drivable or occupied state of grid cells, is desirably extended to that of a semantic occupancy grid, which encodes the presence or absence at each grid location of an object from a set of object classes. In this sense task performed by the system is closely related to a semantic segmentation of the birds-eye-view space. However, in the present disclosure these occupancies are desirably not considered exclusively, as multiple object classes (for example road, pedestrian crossing and cyclist) may simultaneously occupy the same grid cell.

[0024] The present disclosure therefore desirably provides a technique for performing the task of semantic occupancy grid map prediction from images. Furthermore the system of the present disclosure desirably comprises a deep convolutional neural network architecture which is able to predict accurate birds-eye-view maps from monocular images.

[0025] The predicted occupancy grid map may allow for each grid cell of the map to additionally encode geometric information (of the scene shown on the image). In particular, the predicted occupancy grid map may represent simultaneously an elevation map).

[0026] The neural network may be trained to process a monocular image without depth information as input image and to output the occupancy grid map in an orthographic birds-eye-view space. For example, the projection (i.e. the occupancy grid map) may be at an aleatory angle (e.g. projected from image at 90° to a 45° plane).

[0027] The neural network may comprise a feature extractor which is trained to generate a set of (e.g. down-sampled) feature maps containing multi-scale semantic and geometric features from the input image.

[0028] The neural network may comprise a feature pyramid module which takes as input the set of feature maps outputted by the feature extractor, and is trained to up-sample the set of feature maps to provide context to features at higher resolutions, such that distant objects represented by the set of feature- maps have an increased resolution and nearby objects represented by the feature-maps have a larger spatial context.

[0029] The neural network may comprise a dense transformer layer or a stack of dense transformer layers trained to map features from a feature map on the image plane, e.g. from the upsampled feature maps outputted by the feature pyramid module, having a height H and a width W to the occupancy grid map in a birds-eye-view space having a depth Z and a width X. For example, the feature mapping may be achieved by: mapping a column of feature vectors of height H by the dense transformer layer to a stack of Z depth feature vectors via a pair of fully connected layers, applying this transformation convolutionally in the horizontal direction to yield a birds-eye-view feature map in polar coordinates with dimensions Z x W, wherein, in order to obtain a Cartesian feature map, the polar features are resampled by using a predetermined camera geometry to determine a image column u which corresponds to grid cell location (x, z)^T.

[0030] The neural network may comprise a topdown network trained to process feature maps in the birds-eye-view, in particular on the Cartesian feature map outputted by the dense transformer layer, and to predict the occupancy grid map.

[0031] The system may further comprise at least one digital camera, e.g. a monocular camera, configured to obtain the image.

[0032] The present disclosure further relates to a computer implemented method of predicting a map from an image, comprising: inputting the image to a trained neural network, wherein the neural network is predicts a occupancy grid map, which allows for each grid cell of the map to simultaneously encode the presence of multiple object classes.

[0033] The present disclosure further relates to a computer implemented method of training a neural network to predict a map from an image according to the method of the preceding claim, wherein the network is trained using in combination a binary cross entropy loss function and a maximum entropy loss function.

[0034] Accordingly, once the neural network is trained by this method, it is desirably configured to be used in a system as described above. [0035] The maximum entropy loss function may be configured to encourage the neural network to predict reduced confidence (i.e. increased uncertainty) in ambiguous regions of the occupancy grid map, said ambiguous regions being identified based on a predetermined camera geometry and/or based on predetermined occlusion information in the scene captured by the camera.

[0036] In other words, the maximum entropy loss function may only applied to grid cells where the cell occupancies are invalid, either because they fall outside the visible region of the image, or because they are completely occluded by buildings or other large objects.

[0037] For example, the occlusion information may be predetermined based on information originating from an active sensor, e.g. a LIDAR sensor. Said sensor desirably senses the scene visible on the image.

[0038] The present disclosure further relates to a computer program including instructions for executing the steps of a method according to any one of the preceding methods when said program is executed by a computer.

[0039] The present disclosure further relates to a recording medium readable by a computer and having recorded thereon a computer program including instructions for executing the steps of a method according to any one of the preceding methods.

[0040] It is intended that combinations of the above-described elements and those within the specification may be made, except where otherwise contradictory.

[0041] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure, as claimed.

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, and serve to explain the principles thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Fig. 1 shows a block diagram of a system according to embodiments of the present disclosure;

[0044] Fig. 2 shows an exemplary method using a neural network for predicting a map from according to embodiments of the present disclosure. [0045] Fig. 3 shows schematically the neural network used in the method of fig. 2, and

[0046] Fig. 4 shows an exemplary transformation layer according to embodiments of the present disclosure.

DESCRIPTION OF THE EMBODIMENTS

[0047] Reference will now be made in detail to exemplary embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0048] Fig. 1 shows a block diagram of a system 10 according to embodiments of the present disclosure. The system is configured to carry out the method according the present disclosure. In particular, the system may be configured to carry out computer instructions.

[0049] For example the system may comprise a processor 11 (e.g. at least one CPU and/or GPU) and a memory 13 for executing said instructions. The memory 13 may be a non-volatile memory and it may comprise said instructions (or a computer program), e.g. of the method according the present disclosure, which can be executed by the processor 11 for predicting a semantic map from an image. In particular, the memory may store a trained artificial neural network (ANN), which can be executed by the processor 11 to perform the method described below.

[0050] The system 10 may further comprise an acquisition module 12 configured to acquire images (e.g. one or several cameras, in particular only monocular camera(s) for obtaining monocular surround view images of the environment of the system). For example, the system may comprise a plurality of cameras which together obtain a panoramic (e.g. 360°) image of the system environment, in particular without any depth information. Alternatively it may comprise only one rotating camera.

[0051] The system may be part of a robotic system or a vehicle 30. In other words, the system, in particular its acquisition module 12 may be configured to autonomously move.

[0052] Fig. 2 shows an exemplary method using a neural network for predicting a map from according to embodiments of the present disclosure.

[0053] The neural network is thereby schematically presented in fig. 3. [0054] The neural network desirably has a probabilistic occupancy grid framework trained for semantic occupancy grid map prediction from images. Furthermore the system of the present disclosure desirably is able to predict accurate birds-eye-view maps, in particular from monocular images.

[0055] For this purpose the neural network may have a deep Pyramid Occupancy Network (PyrOcNet) comprising a dense transformer layer. The pyramid occupancy network may consist of four main stages:

• A (backbone) feature extractor for generating multi scale semantic and geometric features from the input image (cf. stage SI).

• A feature pyramid module for upsampling low-resolution feature-maps to provide context to features at higher resolutions (cf. stage S2). Said feature pyramid may be FPN inspired, as described e.g. by Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.

• A stack of dense transformer layers for (together) mapping the image- based features into the birds-eye view (cf. stage S3).

• A topdown network for processing feature maps in the birds-eye-view and for predicting the final occupancy grid map (cf. stage S4).

It is noted that the predicted occupancy grid map may not only comprise information regarding occupancy/object presence but e.g. also geometric information (for example the predicted occupancy grid map may represent simultaneously an elevation map). In the following a more specific example of a neural network according to the present disclosure will be described.

[0056]The neural network of the present disclosure is configured to represent the state of the world as a birds-eye-view semantic occupancy grid map. Occupancy grid maps are a type of discrete random field where each spatial location x_t has an associated state m_ir which may be either occupied {mi = 1), or free (m; = 0), cf. :

Alberto Elfes et al. Occupancy grids: A stochastic spatial representation for active robot perception. In Proceedings of the Sixth Conference on Uncertainty in AI, volume 2929, page 6, 1990.

[0057] In practice, the true state of the world is unknown, so one may treat m; as a random variable, and estimate the probability of occupancy p(^mi\^zi-._t), conditioned on a set of observations z_t. The occupancy grid concept may further be extended to that of semantic occupancy grid mapping, in which instead of cells being free or occupied, the state m- represents the presence or absence of an object of class c at a given location. These occupancies are non^¬ exclusive: for example both the road and vehicle classes may coexist at the same location.

[0058] Traditionally in occupancy grid mapping, the conditional occupancy probabilities p( i|z_1:t) are estimated using an inverse sensor model, often a simple function which maps from range sensor readings to occupancy probabilities based on the sensor characteristics. In the present disclosure, the observations take the form of images, which are vastly more complex. It is therefore proposed to train a deep CNN -based model inverse sensor p(m \z_t) = fe (z_t, Xi ) which learns to predict occupancy probabilities from a single monocular input image.

[0059] The objective of the present disclosure is therefore to predict a binary class label at each location on a 2D birds-eye-view image. This scenario bears many similarities to the widely-studied computer vision problem of semantic segmentation. What makes this task particularly challenging however, is the fact that the input and output representations exist within entirely different coordinate systems: the former in the perspective image space, and the latter in the orthographic birds-eye-view space. It is therefore proposed a simple dense transformer layer, which makes use of both camera geometry and fully-connected reasoning to map features from the image to the birds-eye-view space.

[0060] This dense transformer layer is incorporated as part of a deep Pyramid Occupancy Network (PyrOcNet). The pyramid occupancy network consists of four main stages.

[0061] Fig. 4 shows an exemplary (dense) transformer layer (S3) according to embodiments of the present disclosure. The dense transformer layer may first condense the image-based features along the vertical dimension, whilst retaining the horizontal dimension. Then it may be predicted a set of features along the depth axis in a polar coordinate system, which are then resampled to Cartesian coordinates. In more detail the (dense) transformer layer (S3) may comprise: 1. A backbone feature extractor generates multiscale semantic and geometric features from the image.

2. This is then passed to an FPN-inspired feature pyramid which upsamples low-resolution feature-maps to provide context to features at higher resolutions, cf.:

Tsung-Yi Lin, Piotr Doll ^'ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.

3. A stack of dense transformer layers together map the image-based features into the birds-eye view,

4. which are processed by the topdown network to predict the final semantic occupancy grid probabilities

Training

[0062] The neural network may be trained using a combination of two loss functions. The binary cross entropy loss encourages the predicted semantic occupancy probabilities p(mf \z_t) to match the ground truth occupancies in· . Given that the used datasets includes many small objects such as pedestrians, cyclists and traffic cones, it may be made use of a balanced variant of this loss, which up-weights occupied cells belonging to class c by a constant factor a^c\ ,

[0063] Neural networks are however renowned for routinely predicting high probabilities even in situations where they are highly uncertain. To encourage the networks to predict high uncertainty in regions which are known to be ambiguous, one may introduce a second loss, which maximises the entropy of the predictions, encouraging them to fall close to 0.5:

[0064] This maximum entropy loss may be applied only to grid cells which are not visible to the network, either because they fall outside field of view of the image, or because they are completely occluded (see Section 2 for details). One may ignore the cross entropy loss in these regions. The overall loss is given by the sum of the two loss functions:

L total = L xent + XL· uncert (3) where X is a constant weighting factor.

Temporal and sensor data fusion

[0065] The Bayesian occupancy grid formulation provides a natural way of combining information over multiple observations and multiple timesteps using a Bayesian filtering approach [?]. Consider an image observation z_t taken by a camera with extrinsic matrix M_t . One may begin by converting the occupancy probabilities p(m \z_t) into a log-odds representation which conveniently is equivalent to the network's pre-sigmoid output activations. The combined log-odds occupancies over observations 1 to t is then given by from which the occupancy probabilities after fusion can be recovered by applying the standard sigmoid function

The log-odds value l_Q ^c represents the prior probability of occupancy for class c: [0066] To obtain the occupancy probabilities in the global coordinate system, one may resample the output from the network, which predicts occupancies in the local camera-frame coordinate system, into the global frame using the extrinsics matrix M_t, i.e. . This approach may be used both to combine sensory information from a set of surround view cameras, and also to fuse occupancy grids over a e.g. 20s duration sequence of observations.

[0067] Either the cell is occupied ( Cm,) = 1) or free ( (pΐi) = 0). This concept may be further extended to that of semantic occupancy grid maps, where each grid location is associated with a set of probabilities p(m-) indicating the presence or absence of an object of class c at that location (a depth sensor reading) z₇. In the present disclosure, these observations take the form of images, and therefore a natural approach is to train a CNN to predict the occupancy of each grid cell. The resulting CNN may be termed a Deep Inverse Sensor Model (DISM).The DISM may consist of three main components:

[0068] Desirably it is estimated the marginal posterior of class presence for each grid cell p(m[|z), which is referred to as the inverse sensor model. The observations z₇ take the form of images captured from surround-view cameras. Given the highly complex relationship between image features and semantic map cell occupancies, it is proposed to learn this mapping using a convolutional neural network (CNN): the output. Unlike in semantic segmentation however, in the task there is no direct correspondence between input pixels in the image space and grid cells in the birds-eye-view map space. To overcome this, previous methods have used an auto-encoder-style approach which collapses the input image down to a single feature vector, before using a separate decoder to recover the output map dimensions. However this approach means that much of the relative spatial information within the feature maps may be lost. Much of the recent successes of semantic segmentation approaches may be attributed to avoiding aggressive reduction in feature map dimensions, for example by replacing downsampling layers with dilated convolution, or by incorporating skip connections. Estimates, as well as semantic information which will be used to determine the semantic occupancy of each grid cell. To this end it may be adopted a standard ResNet-50 backbone network which is pretrained e.g. on the ImageNet dataset.

Dense transformer layer [0069] One of the fundamental challenges of the occupancy grid prediction task is that the input and output exist in two entirely disparate coordinate systems: the perspective image space and the orthographic birds-eye-view space. To overcome this problem, a simple transformation layer may be introduced, which is depicted in Fig. 4. The objective is to convert from an image plane feature map with C channels, height H and width W , to a feature map on the birds-eye-view plane with C channels, depth Z and width X.

[0070] The dense transformer layer is inspired by the observation that while the network needs a lot of vertical context to map features to the birds- eye-view (due to occlusion, lack of depth information, and the unknown ground topology), in the horizontal direction the relationship between BEV locations and image locations can be established using simple camera geometry. Therefore, in order to retain the maximum amount of spatial information, it may be collapsed the vertical dimension and channel dimensions of the image feature map to a bottleneck of size B, but preserve the horizontal dimension W. One may then apply a ID convolution along the horizontal axis and reshape the resulting feature map to give a tensor of dimensions C x D x W. However this feature map, which is still in image-space coordinates, actually corresponds to a trapezoid in the orthographic birds-eye-view space due to perspective, and so the final step is to resample into a Cartesian frame using the known camera parameters / and u₀.

[0071] Furthermore a set of C features for Z discrete depth values in the birds-eye-view grid is predicted. The resulting feature map is expressed in a polar coordinate system, and so a final resampling stage may be applied to convert the feature map into Cartesian coordinates.

[0072] Accordingly, applying this transformation convolutionally in the horizontal direction yields a birds-eye-view feature map in polar coordinates with dimensions Z x W. To obtain the final Cartesian feature map, the polar features may be resampled by using camera geometry to determine the image column u which corresponds to grid cell location (x, z)^T: [0073] In may be assumed that the cameras are calibrated and therefore the focal length f and pixel offset uo are known. Multiscale transformer pyramid

[0074] As mentioned, the resampling step described above may involve, for a row of grid cells a distance z away from the camera, sampling the polar feature map at intervals of where Dc is the grid resolution and s is the downsampling factor of input feature map with respect to the image. The use of a constant factor for s however is problematic: features corresponding to grid cells far from the camera will be blurred whilst those close to the camera will be undersampled and aliasing can occur. It is therefore proposed to apply multiple transformers, acting on a pyramid of feature maps with downsampling factors s_k = 2 ^k+3,k e {0, ...,4}. The k^th transformer generates features for a subset of depth values, ranging from z_k to z_k-lf where z_k is given by fAx

¾ = — ^sk (⁹)

[0075] The final birds-eye-view feature map is then constructed by concatenating the outputs of each individual transformer along the depth axis.

[0076] One downside of this approach is that at high resolutions the height of the feature maps H_k can become very large, which leads to an excessive number of parameters in the corresponding dense transformer layer. In practice however, the feature maps may be cropped to a height

Ύpiac Ymin

H_k = f (10 s ) k^zk corresponding to a fixed vertical range between y_min and y_max in the world space. This means that the heights of the cropped feature maps stay roughly constant across scales.

[0077] The feature maps are taken from the outputs of each residual stage in the backbone network, from conv3 to conv7. To ensure that the high resolution feature maps still encompass a large spatial context, it may be added upsampling layers from lower resolutions.

[0078] The dense transformer layer may initially predict birds-eye-view features in a polar coordinate system, which is then rectified to a Cartesian system via bilinear interpolation. However this implies the difficult question of how many stages of downsampling to perform in the backbone network before applying the transformer layer.

[0079] If the output features are too low resolution, small, distant objects will be lost. If one may perform too little downsampling, large, nearby objects will lack spatial context. The optimum sampling rate s^* is given by k. The k^th output feature map is generated by a transformer which operates on the k^th layer of the feature pyramid, which corresponds to a downsampling factor of s_k = 2^k+3 (since the top layer of the feature pyramid has 8 x downsampling).

[0080] It has been seen in outdoor driving applications that few objects of interest lie significantly above or below the ground plane, so one may limit the transformers to look only at a sensible range of y-coordi nates in the world space, say between y_min and y_max. One may therefore crop the k^th feature map in the range

Exemplary training details

[0081] For the backbone and feature pyramid components of the network, it may be used a pretrained FPN network, which incorporates a ResNet-50 front-end. The topdown network consists of a stack of e.g. 8 residual blocks, including a transposed convolution layer which upsamples the birds-eye-view features from a resolution of e.g. 0.5m to 0.25m per pixel. For the balanced loss weighting a^c, it may be used the square root of the inverse class frequency, as it has been found that using inverse frequency directly leads to a tendancy to overpredict on small classes. The uncertainty loss weighting l is taken as e.g. 0.001. It may be trained all networks until convergence using SGD with a learning rate of e.g. 0.1, batch size e.g. 12 and a momentum of e.g. 0.9. It may be considered a semantic grid cell occupied if it has an occupancy probability p_i^c>0.5 in accordance with the Bayesian decision rule. However, it is noted that this threshold is arbitrary, and may not necessarily provide the highest score. It may therefore also be reported the mean average precision (mAP) metric, which takes into account all possible probability thresholds

[0082] Throughout the description, including the claims, the term "comprising a" should be understood as being synonymous with "comprising at least one" unless otherwise stated. In addition, any range set forth in the description, including the claims should be understood as including its end value(s) unless otherwise stated. Specific values for described elements should be understood to be within accepted manufacturing or industry tolerances known to one of skill in the art, and any use of the terms "substantially" and/or "approximately" and/or "generally" should be understood to mean falling within such accepted tolerances.

[0083] Although the present disclosure herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure.

[0084] It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims.

Claims

1. A system for predicting a map from an image, comprising: a trained neural network configured to receive the image as input and to predict in response an occupancy grid map, which allows for each grid cell of the map to simultaneously encode the presence of multiple object classes.

2. The system according to the preceding claim, wherein the map predicted from the image is a semantic map, and/or the occupancy grid map is a semantic Bayesian occupancy grid map.

3. The system according to claim 1 or 2, wherein the predicted occupancy grid map allows for each grid cell of the map to additionally encode geometric information.

4. The system according to any one of the preceding claims, wherein the neural network is trained to process a monocular image without depth information as input image and to output the occupancy grid map in an orthographic birds-eye-view space.

5. The system according to claim 1 or 2, wherein the neural network comprises a feature extractor which is trained to generate a set of feature maps containing multi-scale semantic and/or geometric features from the input image.

6. The system according to the preceding claim, wherein the neural network comprises a feature pyramid module which takes as input the set of feature maps outputted by the feature extractor, and is trained to upsample the set of feature maps to provide context to features at higher resolutions, such that distant objects represented by the set of feature-maps have an increased resolution and nearby objects represented by the feature- maps have a larger spatial context.

7. The system according to the preceding claim, wherein the neural network comprises a dense transformer layer or a stack of dense transformer layers trained to map features from a feature map on the image plane, in particular from the upsampled feature maps outputted by the feature pyramid module, having a height H and a width W to the occupancy grid map in a birds-eye-view space having a depth Z and a width X, in particular by: mapping a column of feature vectors of height H by the dense transformer layer to a stack of Z depth feature vectors via a pair of fully connected layers, applying this transformation convolutionally in the horizontal direction to yield a birds-eye-view feature map in polar coordinates with dimensions Z x W, wherein, in order to obtain a Cartesian feature map, the polar features are resampled by using a predetermined camera geometry to determine a image column u which corresponds to grid cell location (x, z)^T.

8. The system according to the preceding claim, wherein the neural network comprises a topdown network trained to process feature maps in the birds-eye-view, in particular on the Cartesian feature map outputted by the dense transformer layer, and to predict the occupancy grid map.

9. The system according to the preceding claim, further comprising at least one digital camera, in particular a monocular camera, configured to obtain the image.

10. A computer implemented method of predicting a map from an image, comprising: inputting the image to a trained neural network, wherein the neural network is predicts a occupancy grid map, which allows for each grid cell of the map to simultaneously encode the presence of multiple object classes.

11. A computer implemented method of training a neural network to predict a map from an image according to the method of the preceding claim, wherein the network is trained using in combination a binary cross entropy loss function and optionally a maximum entropy loss function.

12. The method according to the preceding claim, wherein the maximum entropy loss function is configured to encourage the neural network to predict reduced confidence in ambiguous regions of the occupancy grid map, said ambiguous regions being identified based on a predetermined camera geometry and/or based on predetermined occlusion information in the scene captured by the camera.

13. A computer program including instructions for executing the steps of a method according to any one of the preceding method claims when said program is executed by a computer.

14. A recording medium readable by a computer and having recorded thereon a computer program including instructions for executing the steps of a method according to any one of the preceding method claims