KR102196255B1 - Apparatus and method of image processing and deep learning image classification for detecting road surface damage - Google Patents

Abstract

A road damage detection apparatus is disclosed, wherein the apparatus vertically projects a photographed image so that the photographing direction is vertical when the photographing direction of the photographed image and the ground formed by the photographing direction and the ground are not vertical. And a control module for pre-processing the projected image, setting a region of interest from the pre-processed image, and extracting a candidate damaged region by performing a binarization operation based on a plurality of parameters for the set region of interest, the control module The damage type of the candidate damage area may be predicted based on the previously learned road damage area information obtained by classifying the road damage area by type. Accordingly, a derivative accident due to road damage can be prevented.

Description

Image processing and deep learning image classification device and method for detecting road damage {APPARATUS AND METHOD OF IMAGE PROCESSING AND DEEP LEARNING IMAGE CLASSIFICATION FOR DETECTING ROAD SURFACE DAMAGE}

The present invention relates to a system for detecting road damage based on a location using image processing and deep learning techniques, and more particularly, to detect a damaged area of the road while monitoring a plurality of lanes, and the type and size of the detected damage area. It relates to a device for determining the like and a road damage detection system using the same.

Road damage can be directly related to the safety of the vehicle driver, so it must be monitored accurately and quickly. Crack, spalling, pot hole, etc. are representative examples of road damage, and cracks represent cracks in the road, and spalling is a surface due to a load applied to a place where surface cracks or inclusions are present. It indicates that it is peeled off slowly, and the porthole represents a local hole that occurs on the pavement surface when the asphalt pavement is shared. Potholes may be caused by weakening of the adhesive strength of the asphalt binder aggregate by the penetration moisture. Portholes not only reduce the common life span of the road, but can also cause vehicle damage and traffic accidents. In particular, in the case of a highway capable of driving at a speed of 80 km/h to 110 km/h, the risk of severe pavement damage such as portholes is inevitably increased.

Therefore, the Korea Expressway Corporation, one of the road management agencies, finds pavement damage that needs urgent repair such as cracks, spalling, potholes, etc. through daily and special inspections, and if road damage is found, it is repaired within 24 hours. However, such a management system based on the visual inspection of human resources is expensive, but it is insufficient to detect road damage early.

Accordingly, there is an urgent need for a method of detecting road damage more quickly and accurately.

Meanwhile, the above information is only presented as background information to aid understanding of the present invention. No decision has been made and no argument is made as to whether or not any of the above is applicable as the prior art for the present invention.

Unexamined Patent Publication No. 10-2016-0009729 (published on January 27, 2016)

The present invention has been devised to solve the above-described problem, and an embodiment of the present invention proposes a system for detecting damage to a road.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the following description. I will be able to.

A road damage detection apparatus mounted on a vehicle according to an embodiment of the present invention for realizing the above-described problem includes: a photographing unit for photographing a road; And if the angle formed by the photographing direction and the ground of the photographed image is not vertical, the photographed image is projected vertically so that the photographing direction is vertical, the projected image is preprocessed, and the region of interest is set in the preprocessed image. And a control module for extracting a candidate damage region by performing a binarization operation based on a plurality of parameters for the set region of interest, wherein the control module is based on the previously learned road damage region information by classifying the road damage region by type. Based on the damage type, the damage type of the candidate damage region may be predicted.

More specifically, the device includes a communication unit for communicating with a road management server; And a location information collection unit that collects the location of the device, wherein the control module collects location information of the determined road damage area through the location information collection unit, and receives the location information of the road damage area through the communication unit. It can be transmitted to the road management server.

More specifically, the device further includes a storage unit, and the control module may store still image and moving image information of a road in which the determined road damage area is located in the storage unit.

More specifically, the control module projects the photographed image vertically so that the photographing direction is vertical when the photographing direction of each of the plurality of photographed images and the angle formed by the ground are not vertical, and the projected image is It can be synthesized frame by frame.

A road damage detection method of a road damage detection apparatus mounted on a vehicle according to an embodiment of the present invention includes: photographing a road; If the angle formed between the photographing direction and the ground of the photographed image is not vertical, projecting the photographed image vertically so that the photographing direction is vertical, and preprocessing the projected image; Setting a region of interest in the preprocessed image and extracting a candidate damage region by performing a binarization operation using a plurality of parameters on the set region of interest; And predicting the damage type of the candidate damage area based on the previously learned road damage area information obtained by classifying the road damage area by type.

According to various embodiments of the present invention, the following effects can be derived.

First, by providing a road damage detection system, a road damage area can be detected in real time.

Second, by detecting the damaged area of the road, the safety of road users can be further protected.

Third, since the damaged area of the road is automatically detected by the device instead of by visual detection, detection efficiency can be improved and detection convenience can be improved.

Fourth, as pavement damage during driving is immediately detected and traced, timeliness in decision-making by road managers can be secured.

Fifth, the road damage area is automatically detected in real time, thereby reducing cost.

Sixth, safety accidents of inspectors who visually check road damage can be prevented.

Seventh, multiple lanes can be monitored at the same time, so detection efficiency can be improved.

Eighth, when determining the road damage area, the type of the road damage area can be easily determined by using the learned road damage area for each type.

Ninth, it is possible to calculate the size of the damaged road area by image processing to calculate the amount of repair and to provide a criterion for determining whether to repair immediately.

The effects that can be obtained in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a diagram schematically illustrating a typical situation to which an embodiment of the present invention is applied.
2 is a block diagram showing the configuration of a road damage detection system according to an embodiment of the present invention.
3 is a sequence diagram illustrating a process of detecting a road damage area and a process of transmitting a detection result according to an embodiment of the present invention.
4 to 9 are views for explaining in detail steps of a process of detecting a road damage area according to an exemplary embodiment of the present invention.
10 and 11 illustrate screens displayed by a road damage detection apparatus according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in more detail with reference to the accompanying drawings. However, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

1 is a diagram schematically illustrating a typical situation to which an embodiment of the present invention is applied.

According to FIG. 1, one direction is indicated on a two-lane highway 40. The center line is divided into a center divider 20, and a vehicle 10 equipped with a terminal for detecting road damage on a two-lane highway is driving a specific lane.

There are a first road damage area 30-1 and a second road damage area 30-2 in the traveling direction of the vehicle 10. The first road damage area 30-1 and the second road damage area 30-2 may be cracks, spalling, portholes, or the like, but the type of the road damage area is not limited to the above description.

A terminal that detects road damage mounted on the vehicle 10 may detect the first road damage area 30-1 and the second road damage area 30-2. Specifically, the terminal detecting road damage may detect the first road damage area 30-1 and the second road damage area 30-2 using an image detection method.

That is, a terminal that detects road damage mounted on the vehicle 10 may simultaneously detect road damage regions of a plurality of lanes.

Hereinafter, a road damage detection system 1000 including the apparatus 100 for detecting damage to the road and the road management server 200 described above will be described with reference to FIG. 2. The device 100 for detecting road damage is mounted on the vehicle 10 and used to directly detect the damage on the road, and the road management server 200 processes and manages information received from the road damage detection device 100. I can.

First, the terminal 200 for detecting damage to the road includes a communication unit 110, a location information collection unit 120, a storage unit 130, a photographing unit 140, a display 160, and a control module 170. can do. However, the components shown in FIG. 2 are not essential in implementing the device 100 for detecting damage to the road, so the device 100 for detecting damage to the road described in this specification is the components listed above. It may have more or fewer components than those.

The communication unit 110 is a module that communicates with the road management server 200. The communication unit 110 transmits information (type, size, shape, shape, characteristic information, etc. of the damaged area) and location information on the damaged area generated on the road under the control of the control module 170 to the road management server 200. Can be transmitted.

The communication unit 110 transmits and receives a wireless signal with at least one of a base station, an external terminal, and a road management server 200 on a mobile communication network established according to technical standards or communication methods for mobile communication. In addition, the communication unit 110 may support short-range communication such as Bluetooth and Wi-Fi.

The location information collection unit 120 may collect location information where a road damage area is generated under the control of the control module 170. The location information collection unit 120 is a module for acquiring the location (or current location) of the road damage detection device 100, and representative examples thereof include a GPS (Global Positioning System) module or a WiFi (Wireless Fidelity) module. For example, the road damage detection apparatus 100 may acquire the location of the road damage detection apparatus 100 by using a signal transmitted from a GPS satellite by using a GPS module. As another example, the road damage detection apparatus 100 uses a Wi-Fi module, based on information of a wireless access point (AP) that transmits or receives a wireless signal with the Wi-Fi module, the road damage detection device ( 100) position can be acquired

The storage unit 130 may store various types of information according to the control of the control module 170. The storage unit 130 includes a flash memory type, a hard disk type, a solid state disk type, an SDD type, and a multimedia card micro type. micro type), card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable (EEPROM) programmable read-only memory), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk.

The storage unit 130 may store information related to a road damage area image classifier capable of classifying a road damage area. The storage unit 130 may store a plurality of pre-learned information that has determined the types of a plurality of road damage areas. For example, the storage unit 130 may store a large amount of a road damage area image corresponding to a crack, a road damage area image corresponding to spalling, and a road damage area image corresponding to a porthole as long as the memory capacity allows.

In addition, the storage unit 130 may include a moving storage that is photographed through the photographing unit 140. The mobile storage is connected by USB so that internal data can be moved.

The photographing unit 140 may photograph a road. The photographing unit 140 may perform still image photographing, video photographing, and voice recording. The photographing unit 140 may photograph not only a driving lane but also a lane that is not driving. In addition, the photographing unit 140 may simultaneously monitor several lanes.

The display 160 may display an image captured by the photographing unit 140 and an image processed through the control module 170. The display unit 160 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. display), a 3D display, and an e-ink display.

The control module 170 corresponds to a module that generally controls the above-described components.

Hereinafter, a method of detecting a road damage area by the control module 170 will be described with reference to FIG. 3.

3 is a sequence diagram illustrating a process of detecting a road damage area and a process of transmitting a detection result according to an embodiment of the present invention.

First, the control module 170 receives an image through the photographing unit 140 (a camera). The control module 170 may collect and analyze the photographed image in real time. The image captured through the photographing unit 140 of the control module 170 may be collected using a mobile storage, and in implementation, the image captured through wireless communication or wired communication may be collected.

In addition, the control module 170 may receive and collect location information of the road damage detection apparatus 100 through a Global Positioning System (GPS).

The control module 170 may synchronize the captured image and GPS location information.

Then, the control module 170 performs preprocessing on the input image.

The control module 170 may convert the input image into grayscale and remove noise in the image. To this end, a Gaussian function based on the mean (π) and the standard deviation (σ) may be used, and a mask operation having a size of 5 may be used, but embodiments are not limited thereto.

4 to 9 are diagrams for specifically explaining steps in a process of detecting a road damage area according to an exemplary embodiment of the present invention.

FIG. 4 is a diagram for explaining an orthographic transformation of a captured image, a preprocessing and a region of interest setting, and a multi-parameter method of binarization according to an embodiment of the present invention.

According to FIG. 4, the control module 170 may determine whether the input photographed image is photographed at an angle or vertically to the ground. The control module 170 may store information about an angle formed between the photographing direction and the ground at the time of image capturing together with the captured image. The control module 170 may determine whether the captured image is taken at an angle or perpendicular to the ground based on the stored angle information.

If the image is taken at an angle, the control module 170 may convert the image to be displayed in a direction perpendicular to the ground. The conversion method will be described in detail in FIG. 6 and will be omitted here.

Next, the control module 170 may remove noise from the projected image. That is, the control module 170 may apply a preprocessing technique for removing noise to the converted captured image.

When removing noise, the control module 170 converts the orthogonal projection image into grayscale and applies a median filter to remove noise in the image to remove noise from the image. Here, the median filter obtains an intermediate value of the pixel density in a region around a specific pixel and processes it as a desired pixel density. An advantage of this method is that edge components can be preserved.

Thereafter, the control module 170 may set a region of interest to remove a portion irrelevant to the road damage region. Accordingly, the amount of data can be greatly reduced in classifying the damaged area by a deep learning technique to be performed later.

Thereafter, the control module 170 may perform a multi-parameter-based binarization. It will be described with reference to FIG. 5 together.

The control module 170 may extract feature points of road damage by performing binaryization in a multi-parameter method. For example, when the captured image is a gray scale image, the control module 170 sets a threshold value of the number of pixels and sets a value that exceeds the threshold value to 1 and a value that does not exceed the threshold value to 0 to binarize the image. I can. The control module 170 does not use only one optimized binarization technique, but performs binarization on images several times with different thresholds to detect road damage areas such as dark roads, light roads, asphalt, general national highways, and old roads. It can help extract. Accordingly, feature points can be clearly and accurately extracted.

In the case of the drawing of FIG. 4, various feature points can be found in each binarized image, such as a red display area.

6 shows a method of performing orthographic transformation in a direction perpendicular to the ground when the photographing direction is oblique.

The control module 170 finds a linear component along the lane of the road surface in the image taken at an angle, finds four reference points in a straight line forming a vanishing point where the linear component meets the horizon, and then orthogonal projection ( Orthogonal Projection) transformation can be applied. In this case, a standardized image can be extracted regardless of the shooting angle and height.

In addition, when the above-described orthogonal projection is performed, a damaged area of a road surface may be photographed using a relatively inexpensive photographing apparatus even if an expensive direct-type photographing apparatus is not used.

7 shows a road surface to which the orthogonal projection technique is applied.

8 is a diagram for explaining a method of classifying a candidate damage region in which the multi-parameter method binarization has been performed.

First of all, deep learning Deep learning is a field of machine learning that attempts a high level of abstraction through a combination of several nonlinear transformation methods. In the road pavement damage detection system, a deep learning technique known as convolutional neural networks (CNN) among deep learning methods can be applied to classify road pavement damage images. CNN is a type of multilayer perceptron designed to use minimal preprocessing, and consists of one or several convolutional layers and a general artificial neural network layer on top of them. Due to this structure, input data of a two-dimensional structure is highly utilized and shows high performance in both video and audio fields.

The control module 170 may apply a deep learning model previously learned from road pavement damage data to classify a damage type for an image extracted as a candidate group in image processing, and finally perform a determination on damage.

To this end, a Graphics Processing Unit (GPU)-based CNN algorithm may be applied.

9 illustrates a method of determining a type (class) of a candidate damaged area image according to an embodiment of the present invention.

First, the control module 170 may classify various types (classes) of the road damage area image through the road damage area image classifier. The control module 170 may learn what kind of image of the road damage area in the thousands to hundreds of thousands (S210). The control module 170 may determine the type of the road damage area image through a deep learning technique using a neural network algorithm (S220).

For example, the control module may upgrade an algorithm for determining what kind of road damage area images are in the tens to hundreds of thousands by using a convolutional neural network (CNN) and a sub-sampling technique.

Based on this, the control module 170 sets the extracted road damage area image as a candidate damage area image (S310).

The control module 170 may determine what kind of candidate damage area image is, using a road damage area image classifier. The road damage area image classifier may also be a part of the control module 170.

The control module 170 may predict and determine the type of the candidate damaged area image based on the learned data.

According to the present invention, it can be observed by experiment that the accuracy of determining the type is 95% or more accurate.

Also, the control module 170 may store all information on an object other than the damaged road area. For example, the control module 170 may store lanes, milestones, safety bars, guard rails, etc. of the road surface that are not road damage areas.

10 and 11 illustrate screens displayed by a road damage detection apparatus according to an embodiment of the present invention.

Referring to FIG. 10, spalling on the road can be determined with 99.9% and 98.8% probability.

On the other hand, various road surface breakages appear according to the physical characteristics of the material paving the road surface. Potholes, turtles, and other crack-like phenomena appear mainly in Asphalt Concrete pavement, and spalling occurs mainly in 　concrete pavement. In addition, patching damage may occur when the hole is broken again after repairing the hole in the road.

Portholes and spalling in particular pose a great threat to the safe driving of a vehicle, and if the size of the damaged road area is more than 15cm, damage such as puncture or bending of the wheel shaft may occur when the wheel is removed.

Referring to FIG. 11, the road damage detection apparatus 100 may determine spalling, potholes, cracks, etc. and convert them into information, and store the grade, size, repair quantity information, location information, time information, etc. have.

When determining the road damage area, the grade can be divided into emergency repair group, risk group, etc., and portholes with a size greater than or equal to the critical size, spalling and patching damage are classified as emergency repair groups that require immediate repair. In some cases, it can be classified as a risk group. Here, the critical size may be set to 15 cm, but the embodiment is not limited thereto.

The amount of repair can be measured by volume by calculating the size and depth of the width and height. The size of the width and height can be calculated through vision processing, and the depth needs to be actually measured, but it is input in a specific range so that the expected value can be calculated. A specific range may be input to about 5 to 7 cm, but embodiments are not limited thereto.

The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of materials that affect a machine-readable radio wave signal, or a combination of one or more of them.

In the present specification, the term "system" or "device" encompasses all devices, devices and machines for controlling data, including, for example, a programmable processor, a computer, or multiple processors or computers. The control system may include, in addition to hardware, code that forms an execution environment for a computer program upon request, such as a code constituting a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. .

Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of a compiled or interpreted language or a programming language, including a priori or procedural language, and can be written as a standalone program or module, It can be deployed in any form, including components, subroutines, or other units suitable for use in a computer environment. Computer programs do not necessarily correspond to files in the file system. A program may be in a single file provided to the requested program, or in multiple interactive files (e.g., files that store one or more modules, subprograms, or portions of code), or part of a file that holds other programs or data. (Eg, one or more scripts stored within a markup language document). The computer program may be deployed to run on one computer or multiple computers located at one site or distributed across a plurality of sites and interconnected by a communication network.

On the other hand, computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, such as magnetic disks such as internal hard disks or external disks, magneto-optical disks, and CDs. -May contain all types of nonvolatile memory, media and memory devices, including ROM and DVD-ROM disks. The processor and memory can be supplemented by special purpose logic circuits or incorporated into it.

Implementations of the subject matter described herein include a backend component, such as a data server, or a middleware component such as an application server, or, for example, a web browser or graphic user that allows a user to interact with an implementation of the subject matter described herein. It may be implemented in a front-end component, such as a client computer having an interface, or a computing system that includes all of one or more combinations of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, for example a communication network.

While this specification includes details of a number of specific implementations, these should not be construed as limiting to the scope of any invention or claim, but rather as a description of features that may be peculiar to a particular embodiment of a particular invention. It must be understood. Likewise, certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. Furthermore, although features operate in a particular combination and may be initially described as so claimed, one or more features from a claimed combination may in some cases be excluded from the combination, and the claimed combination may be a subcombination. Or sub-combination variations.

In addition, although the present specification describes the operations in the drawings in a specific order, it is not understood that such operations must be performed in the specific order or sequential order shown in order to obtain a desirable result, or all illustrated operations must be performed. Can not be done. In certain cases, multitasking and parallel processing can be advantageous. In addition, separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the program components and systems described are generally integrated together into a single software product or packaged in multiple software products. It should be understood that you can

As such, this specification is not intended to limit the invention to the specific terms presented. Accordingly, although the present invention has been described in detail with reference to the above-described examples, those skilled in the art can make modifications, changes, and modifications to these examples without departing from the scope of the present invention. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (5)

A photographing unit for photographing the road;
Storage; And
If the angle formed by the photographing direction and the ground of the photographed image is not vertical, the photographed image is vertically projected so that the photographing direction is vertical, and preprocessing in which edge components are preserved in the projected image is performed,
A region of interest to reduce the amount of data is set by removing information on a portion of the preprocessed image that is not related to the damaged area, and a binarization operation based on a threshold value by a plurality of parameters is performed for the set region of interest. It includes a control module for extracting the base candidate damage region,
The control module,
Using a road damage area image classifier, predict the damage type of the extracted candidate damage area,
Based on the predicted damage type and damage threshold size of the candidate damage area, determine whether emergency repair is required,
The road damage area image classifier,
It is learned to classify the damage type of the road damage area using a neural network algorithm by receiving multiple road damage area images
The control module,
A road damage detection device mounted on a vehicle that stores at least one of information of a still image, a moving image, a grade, a size, a repair quantity, a location, and a time of the road where the determined road damage area is located in the storage unit. The method of claim 1,
A communication unit that communicates with the road management server; And
Further comprising a location information collection unit for collecting the location of the device,
The control module,
A road damage detection device mounted on a vehicle that collects location information of the determined road damage area through the location information collecting unit and transmits the location information of the road damage area to the road management server through the communication unit. delete The method of claim 1,
The control module,
When the photographing direction of each of the plurality of photographed images and the angle formed by the ground are not vertical, the photographed image is projected vertically so that the photographing direction is vertical, and the projected images are synthesized for each frame. Road damage detection device. As a road damage detection method mounted on a vehicle,
Photographing the road;
If the angle formed between the photographing direction and the ground of the photographed image is not vertical, projecting the photographed image vertically so that the photographing direction becomes vertical, and performing preprocessing in which edge components are preserved in the projected image;
A region of interest to reduce the amount of data is set by removing information on a portion of the preprocessed image that is not related to the damaged area, and a binarization operation based on a threshold value by a plurality of parameters is performed for the set region of interest. Extracting a base candidate damage region;
Predicting a damage type of the extracted candidate damage area using a road damage area image classifier; And
Determining whether emergency repair is required based on the predicted damage type and damage threshold size of the candidate damage area,
The road damage area image classifier,
It is learned to classify the damage type of the road damage area using a neural network algorithm by receiving multiple road damage area images,
The road damage detection method,
A method for detecting road damage mounted on a vehicle, further comprising storing at least one of information of a still image, a video, a grade, a size, a repair quantity, a location, and a time of the road damage area on which the determined road damage area is located.

Images