WO2016207749A1

WO2016207749A1 - A device and method of detecting potholes

Info

Publication number: WO2016207749A1
Application number: PCT/IB2016/053239
Authority: WO
Inventors: Marthinus Johannes BOOYSEN; Rodney Stephen KROON; Sonja NIENABER; Yusuf KAKA
Original assignee: Mobile Telephone Networks (Proprietary) Limited
Priority date: 2015-06-23
Filing date: 2016-06-02
Publication date: 2016-12-29

Abstract

A device and method of locating potholes is provided. The method includes capturing, using a camera, an image including a road surface having at least one pothole therein. Extracting from the image a portion of the image including the road surface having at least one pothole therein. The road surface portion of the image is then converted to grey scale and a Gaussian filter is applied to the grey scale image. An edge detection algorithm is applied to the image thereby identifying the area within the detected edges as potholes.

Description

A DEVICE AND METHOD OF DETECTING POTHOLES

BACKGROUND OF THE INVENTION

The present application relates to a device and a method for detecting potholes.

Pothoies in road surfaces are mostly caused by water and regular road maintenance is vital to prevent the decaying process. The manner in which a pothole forms is dependent on the type of bituminous pavement surfacing. The volume of traffic and the axle load experienced by the road are example factors that lead to fatiguing of the road surface, resulting in the formation of cracks. These cracks allow water to seep through and mix with the asphalt. When a vehicle drives over this area, this water wilt be expelled through the crack with some of the asphalt, and this will slowly create a cavity underneath the crack. Eventually the road surface will collapse into the cavity, resulting in a visible pothole, if regular road maintenance is neglected, the aforementioned cracks are not repaired before they cause substantial damage to the road.

For road maintenance to take place, it is obvious that the entity responsible for the road in question must know where the pothole or decaying road section is located and an automated process could assist with this. Potholes are also problematic for drivers as they can cause a lot of damage to their vehicles. Currently, there is no device/system available to drivers that would allow them to avoid potholes.

The present invention seeks to address this. SUMMARY OF THE INVENTION

According to one example embodiment, a method of locating potholes, the method including: capturing, using a camera, an image including a road surface having at least one pothole therein; extracting from the image the road surface having at least one pothole therein; converting the road surface portion of the image to grey scale; apply a Gaussian filter to the grey scale image; apply an edge detection algorithm to the image; and identifying the area within the detected edges as potholes.

The method may further include transmitting to a server camera identification data being data which identifies a camera used to capture the visual data and geographic data including the location of the detected pothole.

According to another example embodiment, a device for locating potholes, the device including: a memory for storing images therein; an image capturing device for capturing images including a road surface having at least one pothole therein and for storing the visual data in the memory; and a processor for: extracting from the image the road surface having at least one pothoie therein; converting the road surface portion of the image to grey scale; apply a Gaussian filter to the grey sca!e image; apply an edge detection algorithm to the image; and identifying the area within the detected edges as potholes.

A GPS module for determining the geographic location at which the pothole image was captured and for storing geographic data describing the location in the memory.

A communications module for transmitting the identified location to a server via a communications network.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a biock diagram illustrating an example device to implement the methodologies described herein;

Figure 2 is a biock diagram illustrating example method steps according to the present invention;

Figure 3 illustrates how a Hull algorithm operates; Figure 4 iiiustrat.es an example output from a Canny edge detector algorithm;

Figure 5 shows an example image captured by the camera from

Figure 1; and

Figure 6 shows how the image from Fig 5 is processed according to the present invention.

DESCRIPTION OF EMBODIMENTS

The device and methodology described herein relate to locating potholes in roads along which vehicles travel.

Referring to the accompanying figures, a device for locating potholes includes apparatus 10 used for locating potholes.

In the illustrated example embodiment, the modules of the apparatus 10 which will be described in more detail below form part of a single apparatus located in a motor vehicle. However, it will be appreciated that the apparatus could also be implemented with some of the modules located remotely from the motor vehicle.

In any event, the apparatus 10 includes an image capturing device 12 which may be in the form of a camera. The camera is located in use on or near the windscreen of the motor vehicle and faces forward outside the motor vehicle where it is able to capture images of a road on which the motor vehicle is travelling.

The device 10 includes a memory 14 for storing data therein for the operation of the apparatus, including storing images captured by the camera 12. Thus it will be appreciated that the camera 12 captures images of a road surface and these wiii then be processed to determine if the road surface has at least one pothole therein.

The processing of the image is accomplished by processor 16.

The broad method steps carried out by processor 16 are illustrated in Figure 2.

Firstly, the camera captures the image in step 24 and this is stored in the memory 14, Processor 16 accesses the memory 14 and extracts an image to be processed.

Next, the processor 16 extracts from the image the road surface, step 26. This is done so that the further processing will not be wasted on an area of the image in which the apparatus is not interested.

The processor 16 then converts the road surface portion of the image to grey scale, step 28, and applies a Gaussian filter to the grey scale image step 30.

Finally, an edge detection algorithm is applied to the image, step 32, to identify the area within the detected edges as potholes, step 34.

The apparatus 10 includes a user interface 18 via which warnings are transmitted to the driver. For example, a warning can be displayed to the driver informing them of a detected pothole which may include an audible alarm to obtain the driver's attention.

The above methodology will now be described in more technical detail.

Referring firstly to step 26 - extracting from the image the road surface having at least one pothole therein. This was performed as in preliminary testing on a prototype of the present invention it was found that irrelevant information in a frame, such as foliage, can lead to false positives.

Therefore, it was decided to first extract the entire road surface automatically and scan only the extracted area for potholes. This is reasonable since the only region of interest with regards to potholes, is the road surface directly in front of the vehicle. Additionally, it eases the classification process as ail other objects and foliage outside of the road has been removed from interfering with the classification.

Extraction of the road was achieved by using a small rectangular region of interest within the image boundary just above the hood of the vehicle. This method assumes that this section will always contain a part of the road if the driver maintains a safe following distance from a vehicle in front.

The mean and standard deviation per color channel of this region of interest is calculated and in order to robustly address the issue of road color variation within the image, the road color is modelled as lying between one standard deviation below and above the mean for each channel.

The contours within the frame that describe the specific road color sections are then determined. In a given frame, it is possible that the average color mode! obtained from the selected region of interest does not describe the road accurately for the entire road surface of that frame (especially when large patches of sand are present). Therefore, to extract the road surface even more accurately, a convex hull algorithm was applied to the extracted contours.

A convex hull algorithm constructs a convex contour around all the furthest points of several points of interest.

An example of this algorithm is shown in Figure 3. Each of the indentations of the hand are marked (A-H) and indicate the gaps in the figure with respect to the outlying points. These gaps are included in the convex hull because the convex hull algorithm will connect each of the furthest points together with straight lines and would encompass the entire hand. Similarly, when sections of the road cannot be extracted via its colour, the convex hull aids in extracting an entire section.

The pothole mode! is derived from the assumption that any strong dark edge within the extracted road surface is deemed a pothole edge if it adheres to certain size constraints. One of the characteristics describing the potholes is a large dark shadow area.

At this point, potholes that do not have dark edges and only have different color variations within them like sand or dirt are disregarded and will be studied in future work. The size constraints were obtained using the selection of images withheld for parameter tuning. Any shape of contour that meets these conditions is deemed a pothole by the algorithm.

As described above, the extracted road section image is converted to a grayscale image in step 28 and then to clear up the image and remove noise, a Gaussian filter is applied to the grayscale image in step 30.

A simple differentiation-based edge detection algorithm is then performed on the extracted road surface, step 34.

An example of an edge detection algorithm is a Canny edge detector which produces a black-and-white image.

An example output of this edge detection is shown in Figure 4, where the input photograph is shown on the left and the output from the edge detection is shown on the right.

Unwanted edges, especially around the outer boundaries of the road, are created by shadows of branches and leaves in trees and are more often worsened by sections of light shining through. Additionally, other vehicles on the road also create unwanted edges. It can be seen from Figure 4 that all of the edges detected by the Canny filter are white lines and are depicted on a black background. The dark regions found within a pothole are usually small in the frame, and often a single pothole will not produce a single edge, but many small edges that are not connected together.

Another problem encountered during prototype testing was that the extracted road surface sometimes has sharp and unwanted unconnected edges on its outer boundaries. These are created by the convex hull algorithm and are dependent on the road colour and lighting conditions.

Due to the approximate shape of the road visible to the camera, grass and dirt alongside a road is also sometimes extracted as part of the road section (due to the convex hull algorithm) and can lead to false positives. To solve this problem, it was determined that the unwanted edges close to the outer boundaries can be removed by dilating the Canny output image several times.

Performing dilation on an image increases the area of the lighter pixels. As a result, when dilation is performed, the unwanted edges close to the outer boundaries become absorbed into the outer boundary leaving only the boundary contour visible. Another advantage of this approach is that it can be used to aid the removal of other vehicles in the frame. Other vehicles on the road are normally found close to the outer boundaries of a frame and sufficient dilation would mean that they too are absorbed into the outer boundary.

The last contour detection is then applied to the dilated image to find the potholes within the road section. The contours are filtered and those that do not meet the size constraints of the pothole mode! are discarded. This last step filters out any small defects in the road that are not classified as potholes as well as the larger contours found on the outer boundary of the extracted road. A test of the abovementioned system and methodology is now described.

A GoPro™ camera was connected to the front windscreen of a car with the setting on 0.5 s time iapse mode and it provided footage that required no deblurhng.

The footage was processed off-line on an Intel Pentium core i7 3.5 GHz with 8 GB of RAM. By timing the duration of the program execution, preliminary observations can be made to determine whether or not the algorithm could be deployed in real time. The software used to create the algorithms was Microsoft Visual C++ 2012 with the OpenCV open source library.

A selection of 53 images containing 97 potholes from the newly created pothole image library were selected as input for the algorithm.

The selection included images from various scenarios, such as driving whilst facing the sun, having the sun on the right of the vehicle, and potholes near trees with large shadows. The input footage was obtained while driving at a speed of approximately 40 km/h, which resulted in subsequent frames showing a single pothole at various distances from the vehicle. It was also found that the lines that indicate the lane separation were either faded or not apparent at all. The appearance of potholes and fading of lines usually coincide, since they both arise as a result of poor maintenance. If the lines were more clear they would have created problems with the algorithm and in future work this aspect will be addressed to ensure a more complete real-world solution.

A full-sized example of the footage that was taken from within the vehicle using the camera mounted to the windscreen of the vehicle is given in Figure 5. it can clearly be seen that there is a lot of information within the image (such as foliage) that is not relevant to pothole detection. This full- sized image is then fed into the algorithm which first extracts the road before any potholes are detected.

An exampie of the step-by-step outputs of the algorithm are illustrated in Figure 6 which corresponds to the input image shown in Figure 5.

The top images in Figure 6 show the output of the road extraction algorithm. The extracted road is then converted to a grayscale image and a Gaussian fi!ter is applied to it to yield the next set of images. The Canny edge detector is then applied, yielding the middle image set.

The second last image set is obtained by the dilation process, which clearly increases the area of ail of the white pixels in the image.

Lastly, the contours within a dilated image are found and those within certain size constraints are classified as potholes.

These results are back-projected onto the initial image which cleariy indicates the potholes detected in the footage. it can be seen that there were potholes present as well as an oncoming vehicle. From the dilation process it can be seen how the wipers and the oncoming vehicle are absorbed into the larger outer boundary and are therefore "filtered" out by this process.

The key performance measures for this study are presented in Table 1.

The sample study indicated a precision of 81.8% and recall of 74.4%. It was also found that the algorithm rejected vehicles on the road 80% of cases (5 frames contained vehicles, and a vehicle was classified as a pothole in only 1 frame). It is possible to increase this accuracy by increasing the dilation process by several factors; however, this will lead to a loss of pothole detection, as more potholes likely merge with the outer road contour.

The formulas to calculate the precision and recall are given in Equations 1 and 2 below:

Due to the nature of the algorithm it was also determined that in the event that two potholes are closely spaced together, they would be grouped together and seen as a single contour. This was mostly the case with potholes that were further away from the vehicle. As the vehicle moved closer, the potholes eventually separate far enough from each other that they can be identified as individual potholes.

An important aspect with respect to the applicability of the aigorithm is the time it takes to run the algorithm. Per frame, it was found that it takes approximately 0.148 seconds to process the road extraction algorithm and 0.037 seconds to detect potholes within the extracted road frame. The total time to complete the algorithm is » 0.2 seconds.

This time does not include optimizations at this point and the code was deployed in debug form. The resulting computational time to complete the detection indicates shows that the algorithm can be deployed on a computer on a vehicle driving at a normal speed and warn the driver ahead of time to avoid the pothole.

From the frames, it is evident that the aigorithm will yield accurate TP results for potholes between approximately 2m and 20m ahead of the vehicle. The lower limitation is due to the dilation process as the pothole merges with the lower lines of the outer boundary of the road whilst the upper limitation has to do with visibility of the pothole through the camera lens. In certain instances, the dilation process will also allow potholes to be absorbed into the outer border if a strong shadow is present across the road, like a bridge or tall tree and the pothole is in close proximity (within ~ 1 m) to the shadow. The upper limitation can be adjusted via the Canny filter parameters, but if it is adjusted to include more potholes at a further distance, it will also currently, include more unwanted edges such as small cracks closer to the vehicle that are not classified as potholes.

It will be thus be appreciated that using a single optical camera, potholes can be detected within a range of = 2 m - 20 m and no training is required.

The device 10 also includes a GPS module 20 for determining the geographic location at which the pothole image was captured and for storing geographic data describing the location in the memory.

A communications module 22 for transmitting the identified location to a server via a communications network.

The communications module 22 is also able to receive locations of potholes detected by others such devices transmitted from a central server to the device 10 and in this way the devices 10 form a network of pothole identifying devices.

In one example embodiment, the device 10 is implemented using a mobile telephone with the modules described in Figure 1 belonging to the mobile telephone.

It will be appreciated that mobile telephones which are so called smart phones often include all of these modules and in order to implement the present invention on the mobile telephone, an executable application is downloaded and executed by a processor 16. The executable application operates together with the hardware as described above in order to implement the methodologies described herein.

Thus it wil! be appreciated that the present invention impiements an effective pothole identifying methodology which warns a driver of a pothole in the road in time to allow the driver to take evasive action to avoid the pothole.

Claims

CLAIMS:

1. A method of locating potholes, the method including: capturing, using a camera, an image including a road surface having at least one pothole therein; extracting from the image a portion of the image including the road surface having at !east one pothole therein; converting the road surface portion of the image to grey scale; apply a Gaussian filter to the grey scale image; apply an edge detection algorithm to the image; and identifying the area within the detected edges as potholes.

2. A method according to claim 1 wherein the extracting from the image of the road surface includes: defining a small rectangular region of interest within the image just above a hood of the vehicle; calculating the mean and standard deviation per color channel of this region of interest; calculating a road color as lying between one standard deviation below and above the mean deviation per color channel; and determining the contours within the frame that describe the road color.

3. A method according to claim 2 further inciuding applying a convex hull algorithm to the extracted contours.

4. A method according to any preceding claim further including transmitting camera identification data to a server, the camera identification data being data which identifies a camera used to capture the image and geographic data including the geographic location at which the pothole image was captured.

5. A method according to any preceding claim further including transmitting a warning to a user when a pothole is detected.

6. A device for locating potholes, the device including: a memory for storing images therein; an image capturing device for capturing an image including a road surface having at least one pothole therein and for storing the image in the memory; and a processor for: retrieving the image from the memory; extracting from the image a portion of the image including the road surface having at least one pothole therein; converting the road surface portion of the image to grey scale; apply a Gaussian filter to the grey scale image; apply an edge detection algorithm to the image; and identifying the area within the detected edges as potholes.

7. A device according to claim 6 further including a GPS module for determining the geographic location at which the pothole image was captured and for storing geographic data describing the location in the memory.

8. A device according to claim 7 further including a communications module for transmitting the geographic data to a server via a communications network.

9. A device according to any one of claims 6 to 8 wherein the processor extracting the road surface from the image includes:

defining a small rectangular region of interest within the image just above a hood of the vehicle; calculating the mean and standard deviation per color channel of this region of interest; calculating a road color as lying between one standard deviation be!ow and above the mean deviation per color channel; and determining the contours within the frame that describe the road color.

10. A device according to any one of claims 6 to 9 wherein the processor further applies a convex hull algorithm to the extracted contours.

11. A device according to any one of claims 6 to 10 further including a user interface for transmitting a warning to a user when a pothole is detected