EP2583217A1

EP2583217A1 - Method for obtaining drivable road area

Info

Publication number: EP2583217A1
Application number: EP10727715.4A
Authority: EP
Inventors: José Manuel ÁLVAREZ LÓPEZ; Felipe LUMBRERAS RUÍZ
Original assignee: Centre De Visio Per Computador; Universitat Autonoma de Barcelona UAB
Current assignee: Centre De Visio Per Computador; Universitat Autonoma de Barcelona UAB
Priority date: 2010-06-21
Filing date: 2010-06-21
Publication date: 2013-04-24
Also published as: WO2011160672A1

Abstract

Method for obtaining drivable road area for assisting a driver of a vehicle, comprising: obtaining at least one current pose of the vehicle through a localization system; obtaining a map model of the road from a repository of road maps according to the at least one current pose of the vehicle; obtaining at least one image of the road through at least one vehicle on-board image capturing device having predetermined device parameters; for each obtained current pose of the vehicle, obtaining a projected model of the road by projecting the map model of the road according to the current pose of the vehicle and to the predetermined device parameters; for each obtained image of the road, obtaining a probabilistic map of the image of the road by applying a computer vision method to the image of the road; and obtaining drivable road area by applying a probabilistic analysis method to the at least one projected model of the road and the at least one probabilistic map of the image of the road.

Description

Method for obtaining drivable road area

The present invention relates to a method for obtaining drivable road area for assisting a driver of a vehicle when said driver driving said vehicle running on said road.

The invention also relates to a computer system and a computer program for obtaining drivable road area for assisting a driver of a vehicle when said driver driving said vehicle running on said road, suitable for carrying out such a method.

Furthermore, the invention relates to a system for obtaining drivable road area for assisting a driver of a vehicle when said driver driving said vehicle running on said road.

BACKGROUND ART

Vision-based road detection is an important research topic in different areas of computer vision such as autonomous driving, car collision warning and pedestrian crossing detection. However, detecting the free-road surface using a vision system is very challenging since the road is an outdoor scenario imaged from a mobile platform. Thus, the detection algorithm must be able to deal with continuously changing background, the presence of different objects (vehicles, pedestrian), different environments (urban, highways, off-road), different road types (shape, colour), and different imaging conditions (varying illumination, different viewpoints and weather conditions).

Current vision-based road detection algorithms mitigate these effects analyzing and grouping pixels according to low-level pixel properties such as colour, texture. For instance, some solutions use the HSI colour space to reduce the influence of lighting variations and texture-less descriptors are also used to characterize road areas. However, algorithms based on these low-level visual cues fail under wide lighting variations (strong shadows and highlights among others) and these algorithms show dependency on highly structured roads, road homogeneity, simplified road shapes, and idealized lighting conditions. The performance of these systems is often improved by including constraints such as temporal coherence or road shape restrictions. Temporal coherence refers to average the results of consecutive frames in an image sequence and use this information in the image being analyzed. Road shape restrictions refers to model the shape of the road using previous results and use this model to restrict the potential road areas in the current image. However, these approaches depend on their previous results. Thus, errors are propagated, specially when other vehicles are present in the scene.

The US patent application US 2009/01 18994 A1 discloses a lane recognizing device that comprises an image processing means which performs a process of estimating a lane of a road by processing an image of the road and outputs a result of the process as first lane information; a lane estimating means which performs a process of estimating the lane using a map data of the road and the current position information of a vehicle and outputs a result of the process as second lane information; and an actual lane recognizing means which recognizes an actual lane of the road on the basis of the first lane information and the second lane information. Said lane recognizing device uses prior road knowledge obtained from map data of the road and the current position information of the vehicle for finally estimating a left-hand line and a right-hand line. On the other hand, an image of the road is obtained (through a vehicle on-board camera) and processed to identify an existing left-hand line and a right-hand line on the road. Then, said estimated left-hand line and right-hand line (from map data and position information) are compared with said identified left-hand line and right-hand line (from captured image) to recognize the actual lane of the road. That is to say, the actual lane of the road recognition is based on comparison between estimated and identified lines.

This approach has two main drawbacks. First, the method requires the existence of specific lane markings. This is an important limitation since these markings are not always present especially in urban environments. Second, this approach is based on simple line representations, thus, error tolerance within the comparison between lines is very little. That is, the possibilities of producing an acceptable result are reduced when errors are present in any of the lane estimation processes: lane estimation from map data and vehicle situation, lane estimation from road images. Hence, this solution has low versatility and robustness.

Another approach exploiting prior road knowledge for road detection consists in using general knowledge of the road by learning off-line some aspects of typical road images and applying this knowledge to process new images. This approach consists in using a scene classifier to provide the probability that a road image contains certain road geometry (left turn, straight, t-like junction). These road geometries are learned off-line using training images. Hence, the algorithm relies on visual information and is limited to a finite number of classes. That is, the current road shape to be detected is limited to the shapes in the training images. Thus, the drawback of low versatility and low robustness still remains in this offline learning solution.

SUMMARY OF THE INVENTION

There thus still exists a need for a new method for obtaining drivable road area for assisting a driver of a vehicle, said method based on on-line acquisition of prior road knowledge, not requiring any kind of marks on the road and improving versatility and robustness of the known methods in the state of the art.

The object of the present invention is to fulfil such a need. Said object is achieved with a method for obtaining drivable road area according to claim 1 , a computer program product comprising program instructions for causing a computer to perform the method for obtaining drivable road area according to claim 16, a computer system for obtaining drivable road area according to claim 19, and a system for obtaining drivable road area according to claim 20. In a first aspect, the present invention provides a method for obtaining drivable road area for assisting a driver of a vehicle, comprising:

obtaining at least one current pose of the vehicle through a localization system; obtaining a map model of the road from a repository of road maps according to the at least one current pose of the vehicle;

obtaining at least one image of the road through at least one vehicle on-board image capturing device having predetermined device parameters;

for each obtained current pose of the vehicle:

obtaining a projected model of the road by projecting the map model of the road according to the current pose of the vehicle and to the predetermined device parameters;

for each obtained image of the road:

obtaining a probabilistic map of the image of the road by applying a computer vision method to the image of the road;

obtaining drivable road area by applying a probabilistic analysis method to the at least one projected model of the road and the at least one probabilistic map of the image of the road.

In other words, a road image is obtained through some on-board image capturing device (e.g., a camera). This captured road image refers to the driver's view of the road. Furthermore, a bird's view representation of the road is obtained from a repository of road maps (e.g. maps server or geographic information system) according to the current pose (position and orientation) of the vehicle. The pose of the vehicle is obtained using a localization system (e.g., global positioning system).

A computer vision method based on image properties such as colour or texture is applied to the road image to obtain a probabilistic map image where each pixel value represents the probability of a pixel being road. In addition, the road model is projected according to the pose of the vehicle. Hence, the bird's view of the road is transferred to the driver's view of the road. Furthermore, this projection of the road model incorporates obtaining a probabilistic map where the value of each map point represents the probability of that point being road. In summary, two probabilistic driver's view maps (i.e., comparable) are obtained: one from the onboard image capturing device and the other from the road map and the pose of the vehicle. Then, a probabilistic analysis method (e.g. Bayesian framework) is applied to the two obtained comparable probabilistic maps for obtaining a final probabilistic map representing the drivable road area or free road surface. In relation to the known methods based on comparison of lines, the method of the present invention increases the error tolerance because this method is based on much richer data, since the image of the road and the projected model of the road are modelled for obtaining probabilistic maps, which is much more than two simple lines defining a lane. Moreover, said lines do not exist in all the types of roads, for example, in urban environments wherein the assumption of specific lane markings on the road is a big source of errors. Consequently, the method of the present invention has the advantage of not requiring any kind of marks on the road and improving versatility and robustness of the known methods in the state of the art.

The reference to the driver's view has been made for best understanding of the method, but it should be understood that the driver's view is in fact the view of the image capturing device in order to be able to compare the probabilistic map coming from the road map and the pose of the vehicle to the probabilistic map coming from the image capturing device. Moreover, the image capturing device can be positioned on the rear part of the vehicle car looking backwards. That is, this method can perform in images of the road ahead or behind the vehicle. Hence, the pose of the image capturing device respect to the car is necessary to obtain the projected road model.

The at least one image capturing device may be two or more image capturing devices. If two or more image capturing devices are used, each device is dedicated to obtain specific data that is taken into account in the method. If more than one image is obtained simultaneously from different points in the vehicle (e.g., different cameras placed at different positions), said images can be merged to obtain more accurate data or extra information to generate the probabilistic map associated to the road image. For example, a stereo acquisition system (two or more image capturing devices) provides two or more images at the same time and a depth map. This depth map is an extra image wherein each pixel contains the distance to the object in the scene. Examples of specific data provided by each camera are: monochromatic information (with grey and texture levels), colour information (with colour and texture), depth information (with geometric features), temperature (for MIR and FIR cameras), etc. In each case of said specific data different specific computer vision algorithms may be used. In summary, the main strength of this new approach is the introduction of road priors estimated on-line to improve the performance of vision-based algorithms, through the combination of road priors and vision-based results using probabilistic analysis. This probabilistic analysis yields a simple formulation of prior influence on road detection. Hence, the probabilistic algorithm exploits the inherent diversity of road priors and low-level based algorithms. The former provides a rough detection of the road despite acquisition conditions whereas the latter provides the required accuracy. Either road likelihood and road priors are built on-line to improve the adaptability of the algorithm to current lighting conditions and the presence of other vehicles in the scene.

In preferred embodiments of the invention, obtaining drivable road area by applying the probabilistic analysis method to the at least one projected model of the road and the at least one probabilistic map of the image of the road comprises: obtaining an averaged projected model of the road by averaging the at least one projected model of the road;

obtaining drivable road area by applying the probabilistic analysis method to the averaged projected model of the road and the at least one probabilistic map of the image of the road. In some embodiments of the invention, obtaining the at least one current pose of the vehicle through a localization system comprises:

obtaining a localization pose of the vehicle through the localization system; obtaining the at least one current pose of the vehicle by applying a random sampling based method to the localization pose of the vehicle.

Therefore, it is proposed to use the uncertainty in the localization pose of the vehicle (obtained from the localization device) to infer the on-line road prior. This uncertainty is modelled using a random sampling based method to reduce the computational cost. Random sampling based methods refer to iteratively evaluating a deterministic model using sets of random numbers as inputs. In this way, a set of values within a range are randomly generated from probabilistic distributions to simulate the process of sampling the complete range. Finally, different probabilistic road maps are obtained from said simulation and are averaged to obtain the road prior (i.e., road probability map) for the corresponding image. Preferably, obtaining a projected model of the road by projecting the map model of the road according to the current pose of the vehicle and to the predetermined device parameters also comprises applying the random sampling based method to the pose of the image capturing device, said pose being comprised in the predetermined device parameters. Thus, different possible poses of the image capturing device are simulated in order to use the uncertainty in the pose of the image capturing device to finally infer the projected model of the road. Therefore, different probabilistic projected models of the road are obtained from said simulation and are averaged to obtain the projected model of the road. This approach of modelling the uncertainty increases the error tolerance of the overall method, since no hard restrictions are required by the method. No particular marks on the road are required and it is assumed that the pose of the vehicle provided by the localization system may comprise some error. The method tries to identify points in the image that "seem" to belong to the road and assuming that the real pose of the vehicle is around (not exactly) the pose obtained through the localization system. That is to say, it is assumed that data elements used in the method contain some errors, which are probabilistically modelled and allowed by the method. Some examples of data elements used in the method are: probabilistic map of the image of the road wherein each point of the image is labelled as belonging to the road or not, pose of the vehicle from the localization system, map from the repository of maps, etc. Preferably, the localization pose of the vehicle comprises a localization position of the vehicle and a localization orientation of the vehicle; and obtaining the localization pose of the vehicle through the localization system comprises:

obtaining the localization position of the vehicle through the localization system; obtaining the localization orientation of the vehicle through the localization system.

Then, the method of the invention may obtain the position and the orientation of the vehicle directly from a localization system able to provide both parameters. Alternatively, the localization pose of the vehicle comprises a localization position of the vehicle and a localization orientation of the vehicle; and obtaining the localization pose of the vehicle through the localization system comprises:

obtaining the localization position of the vehicle through the localization system; storing the obtained localization position of the vehicle into a set of consecutive localization positions of the vehicle;

obtaining the localization orientation of the vehicle from the set of consecutive localization positions of the vehicle.

Thus, the method of the invention can obtain the position and orientation of the vehicle even though the localization system is only able to provide the position, because the method is able to obtain the orientation from consecutive positions provided by the localization system. As it is well known, the orientation may be inferred, for example, by joining with a straight line at least two consecutive positions of the vehicle.

Alternatively, the localization pose of the vehicle comprises a localization position of the vehicle and a localization orientation of the vehicle; and obtaining the localization pose of the vehicle through the localization system comprises: obtaining the localization position of the vehicle through the localization system; verifying if the localization system is able to provide the localization orientation of the vehicle;

in case of positive result:

obtaining the localization orientation of the vehicle through the localization system;

in case of negative result:

storing the obtained localization position of the vehicle into a set of consecutive localization positions of the vehicle;

That is to say, the method is able to determine if the localization system is able to provide position and orientation, in which case, both parameters are directly obtained from the localization device. The method can also determine if the localization device is only able to obtain position, in such a case, said parameter is directly obtained from the localization system and the orientation is inferred from consecutive points provided by the localization system. In preferred embodiments, obtaining the map model of the road from the repository of road maps according to the at least one current pose of the vehicle comprises:

obtaining data representing geometry of the road from the repository of road maps according to the at least one current pose of the vehicle;

obtaining the map model of the road from said data representing geometry of the road.

Thus, the method also takes into consideration if the repository of road maps is able to provide complex data representing the geometry of the road, in which case, the map model of the road can be directly obtained from said geometry of the road. Alternatively, obtaining the map model of the road from the repository of road maps according to the at least one current pose of the vehicle comprises:

obtaining data representing consecutive discrete points and type of the road from the repository of road maps according to the at least one current pose of the vehicle;

obtaining a skeleton of the road by applying interpolation to the consecutive discrete points of the road;

obtaining a predetermined road model related to the type of the road;

obtaining the map model of the road from said skeleton of the road according to the predetermined road model related to the type of the road.

Therefore, the method also takes into consideration if the repository of road maps is only able to provide simple data representing points and type of the road, in which case, the map model of the road is obtained by inferring an skeleton of the road from the points produced by the maps repository and modelling said skeleton according to a standard model of the road related to its type. For example, this standard model may comprise the width of the lane and the width of roadsides depending on the road being highway, primary, secondary or residential, etc. Alternatively, obtaining the map model of the road from the repository of road maps according to the at least one current pose of the vehicle comprises:

obtaining data representing the road from the repository of road maps according to the at least one current pose of the vehicle;

verifying if the data representing the road comprises data representing geometry of the road;

in case of positive result:

obtaining the map model of the road from said data representing geometry of the road;

in case of negative result:

verifying if the data representing the road comprises data representing consecutive discrete points and type of the road;

in case of positive result: obtaining a skeleton of the road by applying interpolation to the consecutive discrete points of the road;

obtaining a predetermined road model related to the type of the road;

That is to say, the method is able to determine if the repository of maps is able to provide geometry of the road, in which case, the map model of the road is directly obtained from said geometry. The method can also determine if the repository of road maps is only able to obtain points and type of the road, in such a case, the map model of the road is obtained by inferring an skeleton of the road from the points produced by the maps repository and modelling said skeleton according to a standard model of the road related to its type. For example, this standard model may comprise the width of the lane and the width of roadsides depending on the road being highway, primary, secondary or residential.

Preferably, each current pose of the vehicle comprises a current position of the vehicle and a current orientation of the vehicle; and obtaining the at least one current pose of the vehicle by applying the random sampling based method to the localization pose of the vehicle comprises:

obtaining each current position of the vehicle by applying the random sampling based method to the localization position of the vehicle;

obtaining each current orientation of the vehicle by applying the random sampling based method to the localization orientation of the vehicle.

The current pose of the vehicle is defined in terms of position and orientation of the vehicle. Then, the random sampling based method is applied to both parameters (position and orientation) of the vehicle previously obtained through the localization system. That is, the uncertainty is modelled in terms of the position and orientation of the vehicle by generating a "cloud" of possible positions and orientations around the position and orientation provided by the localization system. In a preferred embodiment of the invention, the method further comprises storing relevant data of the obtained drivable road area in a repository for off-line error analysis and correction. In some embodiments, the method further comprises sending, through a communications network, relevant data of the obtained drivable road area to a server for on-line error analysis and correction.

Relevant data of the obtained drivable road area, that can be stored in a repository or sent to a server for on-line error analysis and correction, may be the final binary map representing the drivable road area and/or discrepancies between the probabilistic map coming from the image of the road and the probabilistic map coming from the projected map of the road. Said final binary map can be compared with all the maps comprised in the "cloud" of projected models of the road, so that the projected model having lowest discrepancy with the final binary map can be considered the best model (corresponding to the best parameters) and the reference for inferring and correcting errors. Then the method of the invention may also comprise reducing the uncertainty in the next iteration of the method by using the best parameters related to the considered best model. The main advantage of this feature is acceleration of the process of drivable road area generation without decreasing the accuracy of the method.

In preferred embodiments, the random sampling based method comprises a Monte Carlo method. An alternative to Monte Carlo may be a full scanning of the possibilities derived from the variables producing uncertainty for obtaining an uniform distribution of the possible cases, but this approach is very time consuming. Another alternative is obtaining an analytic expression of the problem according to the variables producing uncertainty and obtaining the probability formula to be applied.

Monte Carlo methods are a class of computational algorithms that rely on repeated random sampling to compute their results. Monte Carlo methods are often used in simulating physical and mathematical systems. Because of their reliance on repeated computation of random or pseudo-random numbers, these methods are most suited to calculation by a computer and tend to be used when it is unfeasible or impossible to compute an exact result with a deterministic algorithm.

In preferred embodiments of the invention, the probabilistic analysis method comprises a Bayesian Framework.

The method of the invention is based on an approach consisting in combining the on-line priors and any low-level based algorithm using a Bayesian framework. The result is a pixel-wise confidence map depicting the probability of an image pixel being a road pixel. Then, a classifier assigns a road or background label to each pixel depending on its probability value. An alternative to the Bayesian Framework may be a simple intersection of probability masks. Both the Bayesian Framework and/or the intersection of probability masks may be complemented by adding extra sources of data, for example an accelerometer, said extra sources not being directly joined to provide road probability. From an accelerometer it may be obtained an idea of the orientation of the vehicle and, therefore, from the horizon estimated from said sensor and projected to the image it may be determined that the probabilities of finding what is searched under said horizon are higher. To these approaches it could be added previous information obtained in previous iterations of the method, in which case the modelling may be performed by applying a Bayesian network or a Graphical Model instead of the proposed Bayesian estimator (Bayesian Framework).

In some embodiments of the invention, the computer vision method comprises computer vision algorithms based on at least one of among the following approaches: colour analysis, texture analysis, distribution analysis, neighbourhood analysis, monochromatic analysis (with grey and texture levels), depth analysis (with geometric features), temperature analysis (for MIR and FIR cameras), etc. In a second aspect, the present invention relates to a computer program product comprising program instructions for causing a computer to perform the method for obtaining drivable road area for assisting a driver of a vehicle. The invention also relates to such a computer program product embodied on a storage medium (for example, a CD-ROM, a DVD, a USB drive, on a computer memory or on a readonly memory) or carried on a carrier signal (for example, on an electrical or optical carrier signal).

In a third aspect of the invention, it is provided a computer system for obtaining drivable road area for assisting a driver, comprising:

computer means for obtaining at least one current pose of the vehicle through a localization system;

computer means for obtaining a map model of the road from a repository of road maps according to the at least one current pose of the vehicle;

computer means for obtaining at least one image of the road through at least one vehicle on-board image capturing device having predetermined device parameters;

computer means for obtaining, for each obtained current pose of the vehicle, a projected model of the road by projecting the map model of the road according to the current pose of the vehicle and to the predetermined device parameters; computer means for obtaining, for each obtained image of the road, a probabilistic map of the image of the road by applying a computer vision method to the image of the road;

computer means for obtaining drivable road area by applying a probabilistic analysis method to the at least one projected model of the road and the at least one probabilistic map of the image of the road.

In a fourth aspect, the present invention relates to a system for obtaining drivable road area for assisting a driver, comprising:

the computer system for obtaining drivable road area;

at least one vehicle on-board image capturing device for providing at least one image of the road to the computer system for obtaining drivable road area; a localization system for providing at least one current pose of the vehicle to the computer system for obtaining drivable road area;

a repository of road maps for providing road map data to the computer system for obtaining drivable road area, said road map data being for obtaining a map model of the road according to the at least one current pose of the vehicle.

In a preferred embodiment of the invention, the image capturing device is a camera. The camera may be a matrix camera, for example: CMOS or CCD camera depending on the type of sensors; B/W (grey level), colour camera, or a mixed model (some intensity pixels and some colour pixels) depending on the intensity or colour information; linear or non-linear depending on the dynamic range. The camera can also be a time-of-flight camera (TOF), which is a camera that illuminate the scene with an active illumination and are able to measure the distance to each point of the scene apart from acquiring an intensity image. Or an stereo camera that obtains similar information (intensity/colour and depth) by means of two or more cameras.

Further, the camera may be a camera capturing information out of the visible spectrum (infrared, ultraviolet). In this category of cameras, there are normal cameras with a filter of visible cover which are usually called NIR (Near Infrared) cameras, very used in surveillance. Another type of infrared based cameras are MIR (Medium) or FIR (Far) or even TIR (Thermic).

Additionally, the camera may comprise some devices able to generate depth maps similar to previous examples of stereo and TOF camera, with less resolution but with higher accuracy, as radar and lidar sensors.

In some embodiments, the localization system is a Global Positioning System. In preferred embodiments, the Global Positioning System is an Inertial Global Positioning System. Nowadays it seems to make only sense to use a Global Positioning System (GPS) providing only positions (not orientations) because the alternatives are much more expensive. But, the technology can evolve in the short/medium term towards GPS providing position and orientation without significantly increasing the cost of the GPS. Then, as commented before, the related method for obtaining drivable road area may comprise obtaining only position from the GPS, or obtaining position and orientation from the GPS, or even on-line detection of the type of GPS and obtaining only position or obtaining position and orientation depending on the type of GPS.

Some currently known "advanced" (and expensive) solutions for the GPS are: Differential GPS (DGPS) which provides the position but with high accuracy, Real Time Kinetic GPS (RTKGPS) which provides position and orientation even when the GPS connection fails (comprises sensors able to generate the trajectory of the vehicle).

Nevertheless, it is also possible to use normal (and cheap) GPS with cheap particular sensors and combine the information provided by each one of said devices for obtaining position and orientation, even in case of the GPS connection failure. Some of said sensors that may be combined with a GPS are: compass (electronic), accelerometer, gyroscope, tachometer (the own tachometer of the vehicle in charge of measuring the speed could be used), etc. Any current vehicle normally comprises a plurality of accelerometers in its configuration, so said accelerometers could be adapted to combine their measurements with the GPS information for achieving the mentioned goals.

In preferred embodiments of the invention, the repository of road maps is a maps server. Alternatively, the repository of road maps is a Geographic Information System. And alternatively, the maps repository may be installed on the vehicle, in which case no access to a communications network is needed. That is to say, the maps may be accessible through Internet, in which case it is possible to connect and download the required maps in each moment. But it is also possible to have an on-board repository of road maps, in the same way that GPS have their own maps, and using them during the journey. Other alternatives may be based on accessing to specific traffic networks instead of accessing to public networks (e.g. Internet). In the case of traffic networks, the information could be obtained from elements of the road (e.g. traffic signs, sidelights, traffic infrastructures in general) or even from the closer vehicles on the road.

BRIEF DESCRIPTION OF THE DRAWINGS

Particular embodiments of the present invention will be described in the following, only by way of non-limiting example, with reference to the appended drawings, in which:

Figure 1 is a graphical representation of the main system elements and data elements taking part in obtaining projected models of the road, according to an embodiment of the invention;

Figure 2 is a graphical representation of a bird's view of a road map with superimposed points defining the road skeleton obtained smoothing the road trajectory using non-linear interpolation, according to an embodiment of the invention;

Figure 3 is a graphical representation of a map model of the road according to an embodiment of the invention;

Figure 4 is a graphical representation of the four coordinate systems and relations between them for projecting the map model of the road onto the image plane of the camera, according to an embodiment of the invention;

Figure 5 is a graphical representation of a road confidence map (projected model of the road) obtained modelling the uncertainty in camera pose and car position using a Monte Carlo approach, according to an embodiment of the invention; Figure 6 shows some results of projecting the map model of the road according to the current pose of the vehicle and to the predetermined device parameters, according to an embodiment of the invention;

Figure 7 shows some examples of on-line prior results being obtained using a system and applying a method according to an embodiment of the invention.

DESCRIPTION OF EMBODIMENTS

In the following descriptions, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be understood, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known elements have not been described in detail in order not to unnecessarily obscure the present invention. It is also important to note that the accompanying drawings are not drawn to scale.

In relation to system elements, Figure 1 shows a preferred embodiment of the system for obtaining drivable road area of the invention, said system comprising a computer system (1 12) for obtaining drivable road area; at least one vehicle onboard camera (103) for providing at least one image (104) of the road to the computer system (1 12) for obtaining drivable road area; a localization system (102) for providing at least one current pose (101 , 107) of the vehicle to the computer system (1 12) for obtaining drivable road area; a repository of road maps (1 12) for providing road map data (1 13, 108, 109) to the computer system (1 12) for obtaining drivable road area, said road map data (1 13, 108, 109) being for obtaining a map model of the road (1 10, 1 1 1 ) according to the at least one current pose (101 , 107) of the vehicle.

In preferred embodiments, the projected model of the road (105) or road prior is a probability map comprising a bi-dimensional array where for each cell there is the probability of the corresponding input image pixel depicting road surface. It is also preferred the projected model of the road (105) being estimated online using information available in Geographical Information Systems (GIS) acting as the repository of road maps (1 12). In this case, the system comprises means (1 12) for connecting to Internet and accessing to the GIS. GIS are database systems that capture, store and manage geographically referenced information. That is, data which is linked to physical locations. This information describes the world in geographic terms representing objects like rivers, lakes or roads) using simple geometries such as points, poly-lines or polygons. The proper combination of these geometries creates a bird's view map (1 13) of a specific region of the Earth. Further, road information is usually associated with additional attributes such as road name, road type, construction level, vehicle driving direction and number of lanes in each direction.

The computer system (1 12) is able to store and execute a computer program comprising instructions to perform a method for obtaining drivable road area. The Figure 1 also shows the data elements taking part in said method and the relations between said data elements. The method starts obtaining localization information (101 ) provided by a GPS antenna (102). Secondly, a map model (1 10, 1 1 1 ) of the road is obtained from the GIS according to the localization information (101 ) defining the position of the vehicle. Thirdly, at least one image (104) of the road is obtained through the on-board camera (103), said camera having predetermined parameters associated (pose of the camera on the vehicle, focal length, resolution, etc.). Fourthly, a "cloud" of possible positions (107) of the vehicle is obtained by applying a Monte Carlo approach to the position of the vehicle. The "cloud" of possible positions (107) also takes into consideration diverse poses of the onboard camera (103) in order to model the uncertainty in the pose of the camera (including pose of the vehicle).

Then, each map model (1 10, 1 1 1 ) of the road is projected according to the associated current pose (possible pose) of the vehicle and the predetermined parameters of the camera (including possible pose of the camera), said projection producing a projected model (105, 106) of the road for each map model (1 10, 1 1 1 ) of the road. In other words, projecting the road shape at the current vehicle position onto the 2D driver's view (i.e., the image plane of the on- board camera).

Each one of the obtained projected models (105, 106) of the road comprises a bi- dimensional array where for each cell of the projected model (105, 106) there is the probability of the corresponding input image pixel depicting road surface. The diverse projected models (105, 106) of the road are averaged to obtain an averaged projected model (105) of the road. On the other hand, a computer vision method is applied to each of the obtained images (104) of the road to obtain a probabilistic map of the image (104) of the road associated to said image (104) of the road. Each one of the probabilistic maps of the image (104) of the road comprises a bi-dimensional array where for each cell of the image of the road (104) there is the probability of the corresponding input image pixel depicting road surface.

Finally, the averaged projected model (105) of the road and the probabilistic map of the image (104) of the road are inputted to a Bayesian Framework, wherein said two inputs are probabilistically compared (or analyzed) for producing the drivable road area for assisting the driver of the vehicle.

In preferred embodiments of the invention, the localization information (101 ) provided by the GPS antenna (102) only comprises data defining the position of the vehicle, so the orientation is inferred from a set of consecutive positions (or localization positions) of the vehicle. As shown in Figure 2, a skeleton (21 ) of the road is obtained, said skeleton consisting in a set of junctions (points defining the consecutive positions of the vehicle) (22) connected by piecewise continuous lines (24). These line segments (24) provide a rough description of the road trajectory. Thus, non-linear interpolation (23) is applied to improve the accuracy of the road trajectory, finally obtaining a realistic shape of the road by modelling the road trajectory (25). As shown in Figure 3, in preferred embodiments, the map model of the road is obtained from the skeleton (31 ) of the road according to a predetermined road model related to the type of the road, said type of the road being comprised in the localization information (101 ) provided by the GPS antenna (102). In this way, a road model (Figure 3) consisting of a drivable area (33) and two roadsides (32) is used. The width of these parts, road lane W_RL (34) and roadside W_RS (35), may be estimated according to the road attributes comprised in the road map data (1 13, 108, 109) and additional country's national road legislation according to for example Table 1 .

Table 1

The Figure 4 shows four different coordinate systems involved in the projection of the map models of the road: World Coordinate System (WCS), Vehicle Coordinate System (VCS), Camera Coordinate System (CCS) and Image Coordinate System (ICS). The projection of the map model of the road consists in transferring the map model of the road from VCS to the ICS, using WCS as reference. This process can be decomposed in a rigid body translation (t) and rotation (roll, pitch and yaw) between VCS and CCS, and a perspective projection from CCS to ICS. Given these coordinate systems, a set of N points p = [po, p,,...,p/v] defining the road shape (map model of the road) is mapped (projected) from WCS to ICS as follows,

where K is a matrix characterizing the optical, geometric and digital characteristics of the camera. It is used to link the pixel coordinates of an image point with the corresponding coordinates in the camera reference frame (i.e., image plane of the camera). These parameters are fixed for the particular camera being used and can be estimated through calibration. R (roll, pitch and yaw) and t define the orientation (pose) and location of the camera reference frame with respect to the known vehicle position (VCS). The rotation matrix R is decomposed into individual rotations as follows,

where refer to rotations about the respective axis.

Figure 6 shows some results of projecting the map model of the road according to the current pose of the vehicle and to the predetermined device parameters, according to an embodiment of the invention. As shown, the road shape information is properly recovered for each frame at different daytime and in different situations such as soft and hard turns. The column 61 comprises original images from the on-board camera, the column 62 comprises bird's view of the road map and vehicle location, and column 63 comprises projected models of the road.

These results are obtained at expenses of calibrating the camera pose and position Further, there is an inherent error in the estimation of the

vehicle localization due to inaccuracy in the localization information (i.e., information provided by the GPS antenna). Registration techniques may be used to fix these parameters by matching the projected road with some road features extracted from the current image. However, these registration methods are time consuming and they rely on information from a pre-detected road in the current image. Therefore, uncertainty in the camera pose and position and localization information is used to obtain the on-line road prior (projected models of the road). This uncertainty is modelled using a Monte Carlo approach to reduce the computational cost. Monte Carlo is a method for iteratively evaluating a deterministic model using sets of random numbers as inputs. In this way, a set of values within a range are randomly generated from probabilistic

distributions to simulate the process of sampling the complete range. Finally, all different road maps obtained from Monte Carlo simulation are averaged to obtain the road prior (averaged projected model of the road) for the corresponding image, as shown in Figure 5. That is, Figure 5 is a graphical representation of a road confidence map (projected model of the road) obtained modelling the uncertainty in camera pose and car position using a Monte Carlo approach, according to an embodiment of the invention. Several possible camera poses and car positions (52) are simulated according to the map model (51 ) of the road for obtaining projected models (53, 54, 55) of the road.

Then, road detection consists in combining the on-line priors and any low-level based algorithm using a Bayesian framework (for probabilistic analysis). The result is a pixel-wise confidence map depicting the probability of an image pixel being a road pixel. Then, a classifier assigns a road or background label to each pixel depending on its probability value.

In this classical Bayesian framework, the problem of road detection is formulated as a road likelihood function as follows: where P(road\0) is a probabilistic description of the current estimate of the road (i.e., road probability map) given a set of observations O. These observations are low-level features such as pixel intensity values or colour distributions. ^o(0\road) is the road likelihood function for the same set of observations. That is, a probability density function of a pixel being road with respect to that measurement set. Finally, P(road) is the probability of the /-th pixel in the image to belong to the road surface. That is, the road prior.

Once the road probability map P(road\0) is computed, a classifier assigns a road or background label to each pixel according to its probability of being road. In preferred embodiments, the label assignment is performed by direct thresholding the road probability map using a fixed threshold λ. Hence, a pixel belongs to the road class if P(road\0)≥ X. Otherwise, the pixel belongs to the background class. The remaining is estimating the road likelihood function £o(0\road). In preferred embodiments, the likelihood is computed online for every image using small regions on the bottom part of each frame. Thus, the algorithm is highly adaptive and can cope with sudden changes (i.e., lighting variations and shadows). In particular, a normalized histogram is used as an empirical form of probability distribution for a random variable. There are two clear advantages of non- parametric methods over parametric ones (i.e., mixture of Gaussians): they are fast in training and usage and they are independent from the shape of the likelihood function (i.e., data distribution). Experiments validating the algorithm have been undertaken. The goal of these experiments is assessing the improvement achieved when the road prior is combined with existing road detection algorithms.

Quantitative evaluations are provided using pixel-based measures, from which the following error measures are computed: quality 9, detection accuracy DA, detection rate DR and effectiveness F (Table 2). Each of these measures provides a different insight in the performance of a method. Quality takes into account the completeness of the extracted data as well as its correctness. Detection accuracy, also known as precision, is the probability that the result is valid. Detection rate, or recall, is the probability that the ground-truth data is detected. Effectiveness is a single measure that trad es-off the detection accuracy versus detection rate.

Table 2 Experiments have been conducted on different image sequences taken in real- world scenarios (urban and highway) at different daytime and under weather and lighting conditions. Hence, the dataset covers a wide range of extreme situations such as shadows, tunnels, crowed scenarios and direct light source incident to the camera. These images have been acquired using an on-board camera based on the Micron MT9V023 sensor. This is a high dynamic range CMOS sensor of 752 x 480 pixels and 10 bits per pixel. The camera is equipped with a 6mm focal length microlens. The sensor uses Bayer Pattern for capturing colour information. Standard Bayer pattern decoding (bilinear interpolation) is used to obtain a 3- channels colour image (RGB) of 752 x 480 pixels per channel and 10 bits per pixel.

Each acquired image is geo-referenced (synchronized) with GPS information (latitude, longitude and altitude) provided by a standard GPS antenna (Woxter Slim II) using NMEA protocol. This protocol defines a set of self-contained sentences to interface between different electronic equipment. For instance, a GGA sentence from GPS devices provide the 3D location of the device. Each of these locations (latitude and longitude) is converted to Cartesian coordinates (X, Y, Z) using the datum 84 (WGS-84). This geodesic datum is a reference used to describe the localization of points on the Earth's surface.

The road database (repository of road maps) is obtained from OpenStreetMap, which is an opensource database containing geographical information and road attributes in XML format. These attributes comprise features such as type (motorway, path, trunk, primary road, secondary road), name, maximum speed or one or two ways among others. The resolution of the road trajectory is improved interpolating consecutive road segments using cubic interpolation. The road model is parameterized using equivalences in Table 1 (previously shown) for obtaining map models of the road. Finally, the on-line road prior is estimated using the previously described algorithm based on projecting the map models of the road according to the pose of the vehicle and camera. Example road prior results for different road geometries and different lighting conditions are shown in Fig. 7, wherein row 71 comprises road images from the on-board camera, row 72 comprises projected models of the road (road prior) and row 73 comprises superimposition of obtained road prior onto road images from camera. As shown, the road layout is properly recovered despite lighting and weather conditions since the prior does not rely on current observations.

The improvement in performance when road prior is taken into account is evaluated using different measurement sets. In particular, the Hue-Saturation and Intensity (HSI) colour space and a physics-based illuminant invariant colour space are considered. The former is a transformed colour space which is, to a certain degree, sensitive to lighting variations such as shadows and highlights. This colour space has been widely used for road detection and to process generic outdoor scenes under varying illumination. The latter refers to the feature space introduced by Finalyson. This is a gray-level feature space derived from the Lambertian reflection model to minimize the influence of shadows and lighting variations under Planckian light source assumption. This illuminant-invariant colour space has been successfully used for road detection and in surveillance applications. Six different measurement sets are defined from individual channels of these two colour spaces: Then, using these sets six different road likelihood are computed for each frame in the database. These likelihoods are built using the surrounding region of nine seeds placed at the bottom region of the image. In particular, seeds are placed following an equidistant distribution along rows 440 and 460 of the image. The size of the surrounding area is fixed to 1 1 X 1 1 pixels. These road likelihoods are combined with the on-line prior using the probabilistic framework. The result is a confidence map which is binarized using a fixed threshold. Then, connected components (region growing) is applied to the binary image using the set same seeds used to build the road likelihood. Finally, a fill-in holes process using simple mathematical morphology operations is applied to obtain the final result. The thresholds used for binarizing are fixed using an exhaustive learning approach. In this way, a subset of images from the database is processed and evaluated using all possible values within the range of each parameter. The optimal threshold value is the one which maximizes the average effectiveness (F).

Further, for comparison, road likelihoods (in the probabilistic map of the image of the road) are processed without considering the on-line prior. That is, assuming that all the pixels in the image have the same probability of being road pixels. A summary of the results is listed in Table 3 for urban scenarios and in Table 4 for highways. As shown, a higher performance is achieved when low-level properties include the road prior independently of the measurement set being used. Further, !J exhibits the higher performance in both scenarios. This is an expected result since images in the database contain strong shadows. From these results it can be concluded that considering on-line priors improves the performance of current colour based algorithms enabling reliable road segmentation despite lighting conditions and road shape.

Table 3

Claims

1 . Method for obtaining drivable road area for assisting a driver of a vehicle, comprising:

for each obtained current pose of the vehicle:

for each obtained image of the road:

2. Method according to claim 1 , wherein obtaining drivable road area by applying the probabilistic analysis method to the at least one projected model of the road and the at least one probabilistic map of the image of the road comprises:

obtaining an averaged projected model of the road by averaging the at least one projected model of the road;

obtaining drivable road area by applying the probabilistic analysis method to the averaged projected model of the road and the at least one probabilistic map of the image of the road.

3. Method according to any of claims 1 or 2, wherein obtaining the at least one current pose of the vehicle through a localization system comprises:

4. Method according to claim 3, wherein the localization pose of the vehicle comprises a localization position of the vehicle and a localization orientation of the vehicle;

and wherein obtaining the localization pose of the vehicle through the localization system comprises:

5. Method according to claim 3, wherein the localization pose of the vehicle comprises a localization position of the vehicle and a localization orientation of the vehicle;

6. Method according to claim 3, wherein the localization pose of the vehicle comprises a localization position of the vehicle and a localization orientation of the vehicle;

obtaining the localization position of the vehicle through the localization system; verifying if the localization system is able to provide the localization orientation of the vehicle;

in case of positive result: obtaining the localization orientation of the vehicle through the localization system;

in case of negative result:

7. Method according to any of claims 1 to 6, wherein obtaining the map model of the road from the repository of road maps according to the at least one current pose of the vehicle comprises:

8. Method according to any of claims 1 to 6, wherein obtaining the map model of the road from the repository of road maps according to the at least one current pose of the vehicle comprises:

obtaining a predetermined road model related to the type of the road;

9. Method according to any of claims 1 to 6, wherein obtaining the map model of the road from the repository of road maps according to the at least one current pose of the vehicle comprises:

obtaining data representing the road from the repository of road maps according to the at least one current pose of the vehicle; verifying if the data representing the road comprises data representing geometry of the road;

in case of positive result:

in case of negative result:

in case of positive result:

obtaining a predetermined road model related to the type of the road;

10. Method according to any of claims 4 to 9, wherein each current pose of the vehicle comprises a current position of the vehicle and a current orientation of the vehicle;

and wherein obtaining the at least one current pose of the vehicle by applying the random sampling based method to the localization pose of the vehicle comprises: obtaining each current position of the vehicle by applying the random sampling based method to the localization position of the vehicle;

1 1 . Method according to any of claims 1 to 10, further comprising:

storing relevant data of the obtained drivable road area in a repository for offline error analysis and correction.

12. Method according to any of claims 1 to 1 1 , further comprising:

sending, through a communications network, relevant data of the obtained drivable road area to a server for on-line error analysis and correction.

13. Method according to any of claims 3 to 12, wherein the random sampling based method comprises a Monte Carlo method.

14. Method according to any of claims 1 to 13, wherein the probabilistic analysis method comprises a Bayesian Framework.

15. Method according to any of claims 1 to 14, wherein the computer vision method comprises computer vision algorithms based on at least one of among the following approaches: colour analysis, texture analysis, distribution analysis, neighbourhood analysis.

16. A computer program product comprising program instructions for causing a computer to perform the method for obtaining drivable road area for assisting a driver of a vehicle, according to any of claims 1 to 15.

17. A computer program product according to claim 16, embodied on a storage medium.

18. A computer program product according to claim 16, carried on a carrier signal.

19. Computer system for obtaining drivable road area for assisting a driver, comprising:

computer means for obtaining drivable road area by applying a probabilistic 5 analysis method to the at least one projected model of the road and the at least one probabilistic map of the image of the road.

20. System for obtaining drivable road area for assisting a driver, comprising:

the computer system for obtaining drivable road area according to claim 19;

10 at least one vehicle on-board image capturing device for providing at least one image of the road to the computer system for obtaining drivable road area;

a localization system for providing at least one current pose of the vehicle to the computer system for obtaining drivable road area;

a repository of road maps for providing road map data to the computer system 15 for obtaining drivable road area, said road map data being for obtaining a map model of the road according to the at least one current pose of the vehicle.

21 . System according to claim 20, wherein the image capturing device is a camera.

20

22. System according to any of claims 20 or 21 , wherein the localization system is a Global Positioning System.

23. System according to claim 22, wherein the Global Positioning System is an

25 Inertial Global Positioning System.

24. Method according to any of claims 20 to 23, wherein the repository of road maps is a maps server.

30 25. Method according to any of claims 20 to 23, wherein the repository of road maps is a Geographic Information System.