KR20190094005A - Apparatus and method of image processing and deep learning image classification for detecting road surface damage - Google Patents

Abstract

Disclosed is an apparatus for detecting road damage. The apparatus includes: a photographing unit photographing a road; and a control module vertically projecting an image photographed so that a photographing direction becomes vertical when an angle formed by the photographing direction of the photographed image and the ground is not vertical, preprocessing the projected image, setting an interest area in a preprocessed area, performing a binary coded calculation by a plurality of parameters with respect to the set interest area, and extracting an alternative damage area, wherein the control module can predict a damage kind of the alternative damage area based on pre-learned road damage area information classifying the road damage area by type. Accordingly, the present invention is possible to prevent derivation accidents in accordance with road damage.

Description

Image processing and deep learning image classification apparatus and method for road surface damage detection {APPARATUS AND METHOD OF IMAGE PROCESSING AND DEEP LEARNING IMAGE CLASSIFICATION FOR DETECTING ROAD SURFACE DAMAGE}

The present invention relates to a system for detecting road damage based on location using image processing and deep learning techniques. More particularly, the present invention relates to a system for detecting damage to a road while monitoring a plurality of lanes, and the type and size of the detected damage area. And a road damage detection system using the same.

Damage to the road can be directly linked to the vehicle driver's safety and must be monitored accurately and quickly. Cracks, spalling, potholes, etc. are typical examples of road damage. Cracks represent cracks in the road, and spalling is applied to areas with surface cracks or inclusions. Peeling slowly indicates that the porthole represents a local hole in the pavement surface at the time of sharing the asphalt pavement. The pothole may be generated by being weakened by moisture penetrated by the adhesive force of the asphalt binder aggregate. Portholes not only reduce the common life of the road, but can also cause vehicle damage and traffic accidents. In particular, in the case of a highway capable of driving at a speed of 80 km / h ~ 110 km / h, the risk due to severe pavement damage of roads such as portholes is inevitably increased.

Therefore, Korea Expressway Corporation, one of the road management agencies, seeks pavement damage that requires emergency repairs such as cracks, spalling, and portholes through daily inspections and special inspections, and repairs them within 24 hours when road damages are found. However, such a management system by visual inspection of human resources is expensive and is insufficient to detect road damage at an early stage.

Accordingly, there is an urgent need for a method of detecting road damage more quickly and accurately.

On the other hand, the above information is only presented as background information to help the understanding of the present invention. No determination is made as to whether any of the above is applicable as the prior art concerning the present invention, and no claims are made.

Publication No. 10-2016-0009729 (published: 2016.1.27)

The present invention has been made to solve the above problems, an embodiment of the present invention proposes a system for detecting damage to the road.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned above will be clearly understood by those skilled in the art from the following description. Could be.

Road damage detection device mounted on a vehicle according to an embodiment of the present invention for realizing the above object is a photographing unit for photographing the road; And when the photographing direction of the photographed image and the angle formed by the ground are not vertical, project the photographed image vertically so that the photographing direction is vertical, preprocess the projected image, and set a region of interest in the preprocessed image. And extracting a candidate damage area by performing a binarization operation based on a plurality of parameters with respect to the set ROI, wherein the control module includes pre-learned road damage area information that classifies the road damage area by type. Based on this, damage types of the candidate damage areas can be predicted.

More specifically, the apparatus includes a communication unit for communicating with the road management server; And a location information collection unit for collecting the location of the device, wherein the control module collects the location information of the determined road damage area through the location information collection unit and collects the location information of the road damage area through the communication unit. Can be sent to the road management server.

More specifically, the apparatus may further include a storage, and the control module may store the still image and video information of the road where the determined road damage area is located in the storage.

More specifically, when the photographing direction of each of the plurality of photographed images and the angle formed by the ground are not vertical, the control module projects the photographed image vertically so that the photographing direction is vertical, and displays the projected image. You can synthesize frame by frame.

Road damage detection method of the road damage detection device mounted on a vehicle according to an embodiment of the present invention, the step of photographing the road; If the photographing direction of the photographed image and the angle formed by the ground are not vertical, projecting the photographed image vertically such that the photographing direction is vertical, and preprocessing the projected image; Setting a region of interest in the preprocessed image and performing a binarization operation using a plurality of parameters on the set region of interest to extract the candidate damage region; And predicting a damage type of the candidate damage area based on previously learned road damage area information classified by type of road damage areas.

According to various embodiments of the present invention, the following effects may be derived.

First, by providing a road damage detection system, the road damage area can be detected in real time.

Second, by detecting the road damage area, the safety of the users using the road can be further protected.

Third, since the road damage area is automatically detected by the device without visual detection, detection efficiency can be improved and detection convenience can be improved.

Fourth, pavement damage can be immediately detected and historyed while driving, thereby ensuring timeliness in decision making by the road manager.

Fifth, since the road damage area is automatically detected in real time, the effect of cost reduction can occur.

Sixth, the safety accident of the inspector who visually confirmed the road damage can be prevented.

Seventh, monitoring of multiple lanes is possible at the same time, thereby improving the detection efficiency.

Eighth, when determining the road damage area, the type of road damage area can be easily determined by using the learned road damage area by type.

Ninth, it is possible to calculate the repair amount by calculating the size of the road damage area by image processing and provide a criterion for determining whether to repair immediately.

Effects obtained in the present invention are not limited to the above-mentioned effects, and other effects not mentioned above may be clearly understood by those skilled in the art from the following description. will be.

1 is a diagram schematically illustrating a representative situation in which an embodiment of the present invention is applied.
2 is a block diagram illustrating a configuration of a road damage detection system according to an exemplary embodiment.
3 is a sequence diagram illustrating a process of detecting a road damage area and a process of transmitting a detection result according to an embodiment of the present invention.
4 to 9 are diagrams for describing in detail the steps of a process for detecting a road damage area according to an embodiment of the present invention.
10 and 11 show screens displayed on the road damage detection apparatus according to an embodiment of the present invention.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

1 is a diagram schematically illustrating a representative situation in which an embodiment of the present invention is applied.

According to FIG. 1, one direction is shown on a two-lane highway 40. The center line is divided into a central separator 20, and a vehicle 10 equipped with a terminal for detecting road damage on a two-lane highway is driving on a specific lane.

The first road damage area 30-1 and the second road damage area 30-2 are in the traveling direction of the vehicle 10. The first road damage area 30-1 and the second road damage area 30-2 may be cracks, spalling, port holes, or the like, but the type of road damage area is not limited to the above description.

The terminal detecting the road damage mounted on the vehicle 10 may detect the first road damage area 30-1 and the second road damage area 30-2. In detail, the terminal detecting the road damage may detect the first road damage area 30-1 and the second road damage area 30-2 by using an image detection method.

That is, the terminal detecting road damage mounted on the vehicle 10 may simultaneously detect road damage areas of a plurality of lanes.

Hereinafter, referring to FIG. 2, a road damage detection system 1000 including the above-described apparatus 100 for detecting damage to a road and a road management server 200 will be described. The apparatus 100 for detecting damage of the road is mounted on the vehicle 10 and used to directly detect the damage of the road, and the road management server 200 may process and manage information received from the road damage detection device 100. Can be.

First, the terminal 200 for detecting damage to the road includes a communication unit 110, a location information collecting unit 120, a storage unit 130, a photographing unit 140, a display 160, and a control module 170. can do. However, the components shown in FIG. 2 are not essential to implementing the apparatus 100 for detecting damage to the road, and thus, the apparatus 100 for detecting damage to the road described herein is a component listed above. It may have more or fewer components.

The communication unit 110 is a module for communicating with the road management server 200. The communication unit 110 transmits the information on the damage area (type, size, shape, shape, characteristic information, etc.) and location information of the damage area generated on the road under the control of the control module 170 to the road management server 200. Can transmit

The communication unit 110 transmits and receives a radio signal with at least one of a base station, an external terminal, and a road management server 200 on a mobile communication network established according to technical standards or communication schemes for mobile communication. In addition, the communication unit 110 may also support short-range communication such as Bluetooth, Wi-Fi.

The location information collection unit 120 may collect location information on which a road damage area is generated under the control of the control module 170. The location information collecting unit 120 is a module for obtaining a location (or current location) of the road damage detection device 100, and a representative example thereof may be a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when the road damage detection apparatus 100 utilizes a GPS module, the road damage detection apparatus 100 may acquire a location of the road damage detection apparatus 100 using a signal transmitted from a GPS satellite. As another example, when the road damage detection apparatus 100 utilizes a Wi-Fi module, the road damage detection apparatus 100 may be configured based on information of a wireless access point (AP) that transmits or receives a Wi-Fi module and a wireless signal. 100 position can be obtained

The storage unit 130 may store various information according to the control of the control module 170. The storage unit 130 may include a flash memory type, a hard disk type, a solid state disk type, an SSD type, a silicon disk drive type, and a multimedia card micro type. micro type), card type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable A storage medium may include at least one type of a programmable read-only memory (PROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

The storage unit 130 may store information related to the road damage area image classifier that may classify the road damage area. The storage unit 130 may store a plurality of pre-learned information for determining the types of the plurality of road damage areas. For example, the storage unit 130 may store a large amount of the road damage area image corresponding to the crack, the road damage area image corresponding to the spalling, and the road damage area image corresponding to the port hole as long as the capacity of the memory allows.

In addition, the storage unit 130 may also include a mobile storage photographed through the photographing unit 140. The mobile storage may be connected by USB so that internal data may be moved.

The photographing unit 140 may photograph a road. The photographing unit 140 may perform still image shooting, video shooting, voice recording, or the like. The photographing unit 140 may photograph not only a driving lane but also a lane that is not driving. In addition, the photographing unit 140 may monitor several lanes at the same time.

The display 160 may display an image photographed by the photographing unit 140 and an image processed by the control module 170. The display unit 160 may include a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible display). display, a 3D display, or an e-ink display.

The control module 170 corresponds to a module for controlling the above-described components as a whole.

Hereinafter, a method of detecting the road damage area by the control module 170 will be described with reference to FIG. 3.

3 is a sequence diagram illustrating a process of detecting a road damage area and a process of transmitting a detection result according to an embodiment of the present invention.

First, the control module 170 receives an image through the photographing unit 140 and the camera. The control module 170 may collect and analyze the photographed image in real time. An image captured by the photographing unit 140 of the control module 170 may be collected using a mobile storage, and in an implementation, the image photographed through wireless or wired communication may be collected.

In addition, the control module 170 may receive and collect location information of the road damage detection apparatus 100 through a global positioning system (GPS).

The control module 170 may synchronize the captured image with the GPS location information.

Next, the control module 170 performs preprocessing on the input image.

The control module 170 may convert the input image into grayscale and remove noise in the image. To this end, a Gaussian function based on mean (π) and standard deviation (σ) may be used, and a mask operation of size 5 may be used, but embodiments are not limited thereto.

4 to 9 are diagrams for describing in detail the steps of a process for detecting a road damage area according to an embodiment of the present invention.

FIG. 4 is a diagram illustrating an orthogonal transformation, preprocessing and region of interest setting, and a multi-parameter binarization technique of a captured image according to an exemplary embodiment.

According to FIG. 4, the control module 170 may determine whether the input photographed image is photographed at an angle or photographed in a direction perpendicular to the ground. The control module 170 may store information about the photographing direction and the angle formed by the ground at the time of photographing the image together with the photographed image. The control module 170 may determine whether the captured image is photographed at an angle or perpendicular to the ground based on the stored angle information.

If the image is taken at an angle, the control module 170 may convert the image to appear in a direction perpendicular to the ground. The conversion method is described in detail in FIG. 6 and will be omitted here.

Next, the control module 170 may remove noise from the projected image. That is, the control module 170 may apply a preprocessing technique for removing noise to the converted captured image.

The control module 170 may remove the noise of the image by applying a median filter to convert the orthogonal converted image to grayscale and remove noise in the image when the noise is removed. In this case, the median filter obtains an intermediate value of pixel concentration in a region around a specific pixel and processes the pixel at a desired pixel density. An advantage of this method is that edge components can be preserved.

Thereafter, the control module 170 may set a region of interest to remove a part not related to the road damage area. Accordingly, the amount of data can be greatly reduced in classifying a damaged area by a deep learning technique to be performed later.

Next, the control module 170 may perform the multiparameter binarization. It will be described with reference to Figure 5 together.

The control module 170 may extract the feature point of the road damage by performing the binarization of the multi-parameter method. For example, when the captured image is a gray scale image, the control module 170 sets the threshold value of the number of pixels so that the value exceeding the threshold is 1 and the value not exceeding the threshold is 0 to binarize the image. Can be. The control module 170 does not use only one optimized binarization technique, but binarizes an image several times with different threshold values, thereby repairing road damage areas such as dark roads, bright roads, asphalt, general roads, and old roads. It can help to extract. Accordingly, the feature point can be extracted clearly and accurately.

In the case of FIG. 4, various feature points may be found in each binarized image like a red display area.

6 illustrates a method of performing orthogonal transformation in a direction perpendicular to the ground when the photographing direction is oblique.

The control module 170 finds a linear component along the lane of the road surface in an obliquely photographed image, finds four reference points in a straight line forming a vanishing point where the linear component meets the horizon, and then uses the orthographic projection ( Orthogonal Projection) transformation can be applied. In this case, a standardized image may be extracted regardless of the photographing angle and the height.

In addition, when the orthogonal projection described above is performed, the damaged area of the road surface may be photographed using a relatively inexpensive photographing apparatus even if an expensive direct photographing apparatus is not used.

7 illustrates a road surface to which an orthographic technique is applied.

8 is a diagram for describing a method of classifying candidate damage regions in which binarization of a multi-parameter method is performed.

First, deep learning deep learning is a field of machine learning that attempts a high level of abstraction through a combination of several nonlinear transform techniques. The road pavement detection system can be applied to classify pavement damage images by a deep learning technique known as convolutional neural networks (CNN). CNN is a type of multi-layered perceptron that is designed to use minimal preprocessing and consists of one or several convolutional layers and common artificial neural network layers mounted on it. Due to this structure, the input data of the 2D structure is highly utilized, and the performance of both the video and audio fields is high.

The control module 170 may apply a deep learning model previously learned with road pavement damage data to classify the damage type for the image extracted as the candidate group in the image processing, and finally determine the damage.

To this end, a GPU (Graphics Processing Unit) based CNN algorithm may be applied.

9 illustrates a method of determining a type (class) of candidate damaged area images according to an embodiment of the present invention.

First, the control module 170 may classify various types of road damage zone images (classes) through the road damage zone image classifier. The control module 170 may learn what kinds of road damage area images are thousands or hundreds of thousands (S210). The control module 170 may determine the type of the road damage area image through the deep learning method using the neural network algorithm (S220).

For example, the control module may upgrade an algorithm that determines what kind of road damage area image is several hundreds to hundreds of thousands of by using a convolutional neural network (CNN) using a composite product and a subsampling technique.

Based on this, the control module 170 sets the extracted road damage area image as a candidate damage area image (S310).

The control module 170 may determine what kind of candidate damage area image is, by using a road damage area image classifier. The road damage area image classifier may also be part of the control module 170.

The control module 170 may predict and determine the type of the candidate damage area image based on the learned data.

According to the present invention, it can be observed by experiment that the accuracy of determining the type is 95% or more accurate.

In addition, the control module 170 may store all the information about the object that is not the road damage area. For example, the control module 170 may store a lane, a milestone, a safety bar, a guard rail, or the like on a road surface that is not a road damage area.

10 and 11 show screens displayed on the road damage detection apparatus according to an embodiment of the present invention.

Referring to FIG. 10, spalling on a road may be determined with a probability of 99.9% and 98.8%.

Meanwhile, various road surface breakages may appear depending on the physical characteristics of the pavement material. Phenomena such as potholes, turtles, and cracks are found mainly in asphalt concrete pavements, and spalling occurs mainly in wet concrete pavements. In addition, a patching failure that may occur after repairing a hole in a road may be broken again.

In the safe driving of the vehicle, especially portholes and spalling are a big threat, and if the size of the road damage area is more than 15 cm, the wheel may fall out, causing puncture or bending of the wheel axle.

Referring to FIG. 11, the road damage detection apparatus 100 may determine the spalling, the port hole, the crack, and the like, and may store the road damage detection device 100 as a grade, size, repair quantity information, location information, time information, and the like. have.

When determining the road damage area, the grade can be classified into emergency repair group, risk group, etc., and portholes, spalling and patching damages of size larger than the threshold size are classified as emergency repair group requiring immediate repair, In that case they may be classified as a risk group. Here, the threshold size may be set to 15 cm, but the embodiment is not limited thereto.

The repair volume can be measured by volume by calculating the size and depth of the width and length. Vision processing can be used to calculate the size of the horizontal and vertical, and the depth should actually be measured, but can be entered into a specific range to estimate the estimate. The specific range may be input as about 5 ~ 7cm, but the embodiment is not limited thereto.

The computer readable medium may be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of materials affecting a machine readable propagated signal, or a combination of one or more thereof.

As used herein, the term "system" or "apparatus" encompasses all the mechanisms, devices, and machines for controlling data, including, for example, programmable processors, computers, or multiple processors or computers. The control system may include, in addition to hardware, code that forms an execution environment for a computer program on demand, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. .

Computer programs (also known as programs, software, software applications, scripts, or code) may be written in any form of programming language, including compiled or interpreted languages, or a priori or procedural languages. It can be deployed in any form, including components, subroutines, or other units suitable for use in a computer environment. Computer programs do not necessarily correspond to files in the file system. A program may be in a single file provided to the requested program, in multiple interactive files (eg, a file that stores one or more modules, subprograms, or parts of code), or part of a file that holds other programs or data. (Eg, one or more scripts stored in a markup language document). The computer program may be deployed to run on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

Computer-readable media suitable for storing computer program instructions and data, on the other hand, include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, such as magnetic disks such as internal hard disks or external disks, magneto-optical disks, and CDs. It may include all types of nonvolatile memory, media and memory devices, including -ROM and DVD-ROM disks. The processor and memory can be supplemented by or integrated with special purpose logic circuitry.

Implementations of the subject matter described herein may include, for example, a backend component such as a data server, or include a middleware component such as, for example, an application server, or a web browser or graphical user, for example, where a user may interact with the implementation of the subject matter described herein. It may be implemented in a computing system that includes a front end component, such as a client computer with an interface, or any combination of one or more of such back end, middleware or front end components. The components of the system may be interconnected by any form or medium of digital data communication such as, for example, a communication network.

Although the specification includes numerous specific implementation details, these should not be construed as limiting to any invention or the scope of the claims, but rather as a description of features that may be specific to a particular embodiment of a particular invention. It must be understood. Likewise, certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, while the features may operate in a particular combination and may be initially depicted as so claimed, one or more features from the claimed combination may in some cases be excluded from the combination, the claimed combination being a subcombination Or a combination of subcombinations.

In addition, although the drawings depict operations in a particular order, it is to be understood that such operations must be performed in the specific order or sequential order shown in order to obtain desirable results or that all illustrated acts must be performed. Can not be done. In certain cases, multitasking and parallel processing may be advantageous. Moreover, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products. Should understand that

As such, this specification is not intended to limit the invention to the specific terms presented. Thus, while the present invention has been described in detail with reference to the above examples, those skilled in the art can make modifications, changes, and variations to the examples without departing from the scope of the invention. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims (5)

Shooting unit for photographing the road; And
If the photographing direction of the photographed image and the angle formed by the ground are not vertical, project the photographed image vertically so that the photographing direction is vertical, preprocess the projected image,
A control module for setting a region of interest in the preprocessed image and extracting a candidate damage region by performing a binarization operation using a plurality of parameters on the set region of interest;
And the control module predicts a damage type of the candidate damage area based on previously learned road damage area information that classifies road damage areas by type. The method of claim 1,
Communication unit for communicating with the road management server; And
Further comprising a location information collecting unit for collecting the location of the device,
The control module,
And collect the determined location information of the road damage area through the location information collection unit, and transmit the location information of the road damage area to the road management server through the communication unit. The method of claim 1,
Further comprising a storage unit,
The control module,
And storing the still image and video information of the road where the determined road damage area is located in the storage unit. The method of claim 1,
The control module,
When the photographing direction of each of the plurality of photographed images and the angle formed by the ground are not vertical, the photographed image is vertically projected so that the photographing direction is vertical, and the projected images are synthesized frame by frame. Road damage detection device. Photographing the road;
If the photographing direction of the photographed image and the angle formed by the ground are not vertical, projecting the photographed image vertically so that the photographing direction is vertical, and preprocessing the projected image;
Setting a region of interest in the preprocessed image and performing a binarization operation using a plurality of parameters on the set region of interest to extract the candidate damage region; And
And predicting a damage type of the candidate damage area based on previously learned road damage area information classified by type of road damage areas.

Images