KR102119033B1 - Apparatus and method for providing road marking information using image data and vibration - Google Patents

Abstract

Provided are an apparatus for providing road surface marking damage information of a road using an image and a method thereof. The apparatus for providing road surface marking damage information of a road using an image comprises: a camera unit mounted on a vehicle and photographing a road on which the vehicle is driving to output a road surface image; a location sensing unit providing location information of the vehicle; and a device processor determining whether the visibility of a road marking marked on the road is damaged based on the road surface image, and the degree of damage, and providing the road surface image in which the degree of damage is determined and information on a location in which a road surface image.

Description

Apparatus and method for providing road marking information using image data and vibration}

The present invention relates to an apparatus and a method for providing road surface damage information using an image, and more specifically, a road using an image that can be provided by extracting a road surface mark from an image of a road and determining a damaged state It relates to an apparatus and method for providing information indicating damage to the road surface.

Road marking refers to a sign that informs the road's users with signs, letters, or lines on the road surface for the purpose of road traffic safety.

Road markings are largely divided according to color and shape.

First, in the case of color, yellow indicates a central line, road central obstacle among road obstacles, no parking, prohibition of parking, safety zone, and blue indicates a lane for exclusive use of buses and lanes for multi-passenger vehicles. Red is indicated by the border of the speed limit sign installed in the child protection area or residential area, and the normal white color is outside the above standard. Stop).

In the case of the shape, the solid line means that the lane cannot be changed, the two solid lines (double line) mean that the absolute lane cannot be changed, and the dotted line means that the lane can be changed.

In the case of an autonomous vehicle, it is a key road infrastructure that automatically recognizes the lane of the driving lane during road marking by using a front sensor, so that the autonomous vehicle recognizes the lane well and provides adequate luminance and retroreflective performance. It is most important to keep it.

In addition, regardless of the time of day or weather conditions, the visibility of the lane must be maintained, and the lane must be properly managed so that the driver or autonomous vehicle can recognize the lane well. Traffic accidents caused by damaged lanes, especially in the era of self-driving cars, require preemptive response to lane management.

Currently, the responsibility of managing lanes lies with road managers who have legal powers, and road management agencies must maintain the performance of luminance, etc., to maintain the proper state of lanes, but in reality, compared to wide roads, a small number of manpower is required to display numerous roads. In general, it is difficult to check for damage to the linear lane from time to time, and it is common to manage the lane inspection by visual observation by manpower or manual measurement by a measuring instrument.

Accordingly, there is a need for a technique capable of more accurately and automatically collecting and determining the state of the road surface marking.

Domestic registered patent No. 10-1543342 (2015.08.04. registered)

The technical problem to be achieved by the present invention in order to solve the above-described problem is to obtain a road surface image in real time through a driving vehicle to determine a road surface display having poor visibility, that is, a damaged road surface display, and a damaged road surface display is determined. The present invention is to provide an apparatus and method for providing road marking damage information using an image capable of providing local location information and road marking information.

The problem of the present invention is not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above-described technical problem, according to an embodiment of the present invention, an apparatus for providing road surface damage information using an image is a camera unit mounted on a vehicle and photographing a driving road to output a road surface image ; A location sensing unit providing location information of the vehicle; And based on the road surface image, determine whether the visibility of the road surface displayed on the road is damaged and the degree of damage, and provide the road surface image and the location information on which the road surface image was taken. It includes a device processor.

The device processor extracts a road surface display by inputting a road surface image input in real time from the camera unit to a road surface extraction model previously learned based on artificial intelligence, and displays a road surface using luminance of the extracted road surface display It judges whether it is damaged or not.

The apparatus processor may include a road surface display extraction unit that extracts a road surface display surrounded by an outer surface by applying the road surface extraction model to the road surface image, and extracts an internal damaged area of the extracted road surface display; A luminance calculator configured to calculate a luminance average of the entire road surface display surrounded by the outline and a luminance average of the internal damaged area; It includes; and a determination unit for determining whether the road surface display is damaged and the degree of damage from the difference between the calculated average luminance of the entire road surface display and the average luminance of the internal damaged area.

The road surface image is converted into a gray image including luminance information, an interest region including a road surface display is set in the converted gray image, and the set region of interest is input to the processor as input data of the road surface extraction model. It further includes an image processing unit to transmit.

The image processing unit, after dividing the road surface image input from the camera unit into three horizontal lines, extracts an analysis area corresponding to the lower 1/3 area, and extracts an area of interest corresponding to a predetermined position among the analysis areas, The camera is installed to photograph along the driving direction from the road surface of the road closest to the front surface of the vehicle, but is installed so that the vanishing point of the road surface horizontal line in the captured image is located within 2/3 of the road surface image.

Meanwhile, according to another embodiment of the present invention, a method of providing road surface damage information using an image of an electronic device includes: (A) the electronic device photographs the road with a camera mounted on a vehicle and outputs a road surface image. step; (B) the electronic device sensing the location information of the vehicle using a GPS module; And (C) the electronic device determines whether the visibility of the road surface displayed on the road is damaged and the degree of damage based on the road surface image output in step (A), and whether the road display is damaged and the degree of damage is determined. And providing a road surface image and location information on which the road surface image was captured.

In the step (C), a road surface image input in real time from the step (A) is input to a road surface extraction model previously learned based on artificial intelligence to extract a road surface display, and the luminance of the extracted road surface display is extracted. Use to determine whether or not the road marking is damaged or not.

In step (C), (C1) the electronic device applies the road surface extraction model to the road surface image, extracts a road surface display surrounded by the outer surface, and extracts an internal damaged area of the extracted road surface display. ; (C2) the electronic device calculating a luminance average of the entire road surface display surrounded by the outer edge and a luminance average of the internal damaged area; And (C3) determining, by the electronic device, whether the road surface is damaged or not, from the difference between the calculated average brightness of the entire road surface display and the average of the luminance of the internal damaged area.

Before the step (C), (D) the electronic device converts the road surface image output in the step (A) into a gray image including luminance information, and includes a road surface display in the converted gray image. And setting the region of interest, and outputting the set region of interest as the input data of the road surface extraction model in the step (C).

In the step (D), after dividing the road surface image input from the camera unit into three horizontal lines, an analysis area corresponding to the lower 1/3 area is extracted, and a region of interest corresponding to a predetermined position in the analysis area is set. In addition, the camera is installed to photograph along the driving direction from the road surface of the road closest to the front surface of the vehicle, but the vanishing point of the road surface horizontal line in the captured image is installed to be located within 2/3 of the road surface image. .

According to the present invention, by implementing a technique for automatically collecting road marking information from a luminance image, it is possible to easily grasp the damaged state of the road marking, and at this time, the various raw data collected is databased and linked with related organizations to manage roads. Smart service can be provided.

According to the present invention, by obtaining a road surface image in real time through a driving vehicle, the shape or luminance of the road surface display is analyzed and the degree of visibility is determined to determine the state of the road display more accurately, thereby increasing the reliability of the determination result. .

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram showing a road surface extraction model generation server according to an embodiment of the present invention;
Figure 2 is a view showing a road surface image of one frame to explain the operation of extracting the analysis area from the road surface image,
3 is a view showing some of the types of road markings,
Figure 4a is an exemplary diagram for explaining the labeling operation of the pre-processing unit,
4B is a view showing an example before and after labeling of a normal road marking, and before and after labeling of a damaged road marking.
FIG. 5 is a block diagram showing an apparatus for providing damage to road surface information according to an embodiment of the present invention;
6 is a view showing a road damage information providing system according to another embodiment of the present invention,
7 is a flowchart illustrating a method for generating a road surface extraction model in the road surface extraction model generation server according to an embodiment of the present invention;
8 is a flowchart for explaining a method for providing road damage information in an apparatus for providing damaged road information according to an embodiment of the present invention;
9 is a diagram illustrating an architecture for road surface damage information according to an embodiment of the present invention described above with reference to FIGS. 1 to 8.

The above objects, other objects, features and advantages of the present invention will be readily understood through the following preferred embodiments related to the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided to ensure that the disclosed contents are thorough and complete and that the spirit of the present invention is sufficiently conveyed to those skilled in the art.

In this specification, when an element, component, device, or system is mentioned to include a component consisting of a program or software, the element, component, device, or system, even if not explicitly stated, is the program or software. It should be understood that includes hardware (eg, memory, CPU, etc.) or other programs or software (eg, drivers required to run the operating system or hardware) necessary to execute or operate.

Also, it should be understood that an element (or component) may be implemented in software, hardware, or in any form of software and hardware, unless specifically stated in the implementation of any element (or component).

In addition, the terminology used herein is for describing the embodiments and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein,'comprises' and/or'comprising' does not exclude the presence or addition of one or more other components.

Hereinafter, with reference to the accompanying drawings, a specific technical content to be carried out in the present invention will be described in detail. It should be noted that, in some cases, parts that are commonly known in describing the invention and that are not significantly related to the invention are not described in order to prevent chaos from coming out for no reason in describing the invention.

Each configuration of the servers 100, 500, or devices 200, 400 shown in FIGS. 1, 5, and 6 indicates that they may be functionally and/or logically separated, and each configuration must be separately The average expert in the technical field of the present invention will be able to easily infer that it does not mean that it is classified as a physical device or written in a separate code.

In addition, the road surface extraction model generation server 100, the road surface damage information providing device 200, the service providing server 300, the information collecting device 400 according to an embodiment of the present invention, and the road surface damage information provided The server 500 may be, for example, a desktop PC (Personal Computer), a server, a laptop PC (Laptop PC), a netbook computer (Netbook Computer), or one of all electronic devices capable of installing and executing a program such as a smartphone. have.

1 is a block diagram illustrating a road surface extraction model generation server 100 according to an embodiment of the present invention.

The road surface extraction model generation server 100 illustrated in FIG. 1 applies a plurality of road surface images to a learning (for example, deep learning) algorithm to extract poor visibility, that is, a damaged road surface on an actual road. A road surface extraction model can be trained and generated.

To this end, the road surface extraction model generation server 100 includes a user interface unit 110, an input unit 120, a database 130, a pre-processing unit 140, a server memory 150, and a server processor 160. Can be.

The user interface 110 is a device for interfacing between the user and the server 100, and receives a user command and transmits it to the server processor 160, or displays a result processed by the server processor 160 on the screen. have.

The input unit 120 may include a communication circuit that receives road road videos from a plurality of vehicle devices (not shown) in a wired or wireless communication method, or may receive a plurality of road road videos stored in a portable memory (not shown). The road surface video includes a plurality of road surface images (ie, frames), and each road road image may be an RGB image including luminance data and RGB data. In addition, one road surface image may be a frame photographed such that the horizontal line of the road surface, that is, the vanishing point is located in the 2/3 area, as shown in FIG. 2.

The database 130 may collect and store a plurality of road surface videos or frame road surface images obtained from vehicle devices or a portable memory through the input unit 120.

In addition, the database 130 stores the region of interest (ROI) labeled for each type or type of road marking in the pre-processing unit 140, and the labeled ROI includes normal road marking or damaged road marking. can do.

Road markings displayed on the road may be classified into lane and road marking symbols as shown in FIG. 3.

Referring to FIG. 3, the shape of the lane is divided into a dotted line, a solid line, and a double solid line, and the color of the lane is divided into white, yellow, blue, and red. In addition, the road marking symbol provides various information for road traffic safety such as uphill slope, concession, and pedestrian crossing notice.

In the case of a lane, the database 130 may store ROIs labeled for each lane type (ie, an outer contour and an inner contour), and selectively map and store the color of the lane to a corresponding ROI. In addition, in the case of the road marking symbol, the database 130 may configure a DB by storing ROIs labeled for each type of road marking symbol (outer contour and inner contour).

The pre-processing unit 140 image-processes the ROI extracted from the collected road surface images.

FIG. 2 is a diagram illustrating a road surface image of one frame to describe an operation of extracting an ROI from the road surface image.

Referring to FIG. 2, when a user selects a road surface image to be used to generate a road surface extraction model from a road surface video by operating the user interface unit 110, the preprocessor 140 RGB-Y the selected road surface image. By converting, a gray image including luminance data is generated, and a luminance value is calculated for each pixel in the gray image. At this time, the pre-processing unit 140 equally divides the gray image of one frame into horizontal lines, and then extracts the area corresponding to the lower third (that is, the area below the vanishing point) as an ROI, and extracts the ROI. Can be displayed on the screen. When the location, sharpness, etc. of the road marking included in the extracted ROI is inappropriate to learn to generate a road marking extraction model, the user may reset the size of the extracted ROI from the gray image.

4A is an exemplary diagram for explaining the labeling operation of the preprocessing unit 140.

Referring to FIG. 4A, when a ROI of a vanishing point or less is set by a user among gray images including luminance data, the user crops the ROI displayed on the screen using the user interface 110 and preprocesses In operation 140, only road surface markings such as lanes may be displayed on the cropped ROI.

The pre-processing unit 140 may extract an area corresponding to the road marking existing in the ROI along the outline, and additionally extract an inner outline inside the extracted outline. In addition, the pre-processing unit 140 may label the ROI in the form of the extracted outline and inner outline to display the damage location and size of the road marking.

The extraction of the inner contour or the inner contour area surrounded by the inner contour may mean that the road marking is damaged. Therefore, when there is no damaged part on the road surface display in the ROI, the pre-processing unit 140 may label the ROI in the form of an extracted outline.

In addition, if there is a damaged area in the road surface display included in the ROI, the pre-processing unit 140 allows the user to differently display the area corresponding to the road surface display according to any one of the type, size, shape, or color of the road surface display. You can also perform labeling. In addition, when there is no damaged area in the road surface display included in the ROI, the pre-processing unit 140 displays the road surface in a normal state, and the area corresponding to the road surface display is differently displayed according to any one of type, size, shape, or color. Labeling can be performed.

4B is a view showing an example before and after labeling of a normal road marking, and before and after labeling of a damaged road marking.

Referring to FIG. 4B, in the case of a normal lane, there is no damaged area in a portion corresponding to the lane among gray images. Therefore, when the ROI is set in the gray image, the pre-processing unit 140 extracts the outer contour (a) of the lane from the set ROI, and then extracts the inner contour, and since there is no extracted inner contour, the ROI is the shape of the outer contour. It can be labeled with a normal road marking.

In addition, in the case of a damaged lane, when the ROI is set in the gray image, the pre-processing unit 140 extracts the outer contour (a) of the lane from the set ROI and then extracts the inner contour (b), and extracts the ROI from the outer contour (a). And it can be labeled with a damaged road surface marking in the form of an inner contour (b).

The server processor 160 or the pre-processing unit 140, which will be described later, generates ROIs including road markings labeled in different colors and/or shapes by user selection, that is, a label data set to generate a database ( 130). At this time, the server processor 160 or the pre-processing unit 140 may cause the mapping information to be stored in the corresponding labeled ROI when the damaged information (or damage class) and the color of the lane are further input by the user.

Also, the pre-processing unit 140 may increase the number of input data for learning by adjusting at least one of the brightness (B, brightness), contrast (C, contrast), and orientation (O, orientation) of the labeled ROI. Accordingly, a number of different label data can be generated from one ROI, which can be reflected in learning or deep learning of various road surface images that may change depending on the environment, such as weather and time when shooting an actual road. Can be.

The server memory 150 may include volatile memory and/or non-volatile memory. In the server memory 150, for example, in order to implement and/or provide operations and functions provided by the road surface extraction model generation server 100, commands or data related to the components 110 to 160, one The above programs and/or software, operating systems, and the like may be stored.

The program stored in the server memory 150 may include a program for implementing a road surface extraction model. The road marking extraction model is a program for extracting the damaged road marking and predicting its type by applying the collected road surface images to the learning algorithm. The learning algorithm may be an artificial intelligence algorithm, specifically, a deep learning algorithm (eg, DNN, RNN, CNN, SVM, etc.).

In the case of deep learning, it is most important to acquire training data, and for this purpose, as described in the pre-processing unit 140, ROI is set and cropped in one entire image frame, and the location, size, or breakage of the broken dataset A labeling operation to display colors is performed, and a learning data set can be constructed by labeling with the same criteria as possible.

The server processor 160 executes one or more programs stored in the server memory 150 to control the overall operation of the server 100. For example, the server processor 160 may execute the prediction model generation program stored in the server memory 150 to learn and generate a road surface extraction model. To this end, the server processor 160 may include a model generator 162.

The model generation unit 162 learns a plurality of ROIs labeled according to the type or shape of the road surface among the road surface images based on deep learning to determine whether the road surface is damaged, that is, a road surface extraction model (that is, an AI inference model or Interface model). The model generator 162 may generate a road surface extraction model by performing a learning algorithm using the ROI selected by the user as input data and the labeled ROI corresponding to the region of interest as output data. The generated road marking extraction model may be installed and executed on the road marking corruption information providing apparatus 200 or the road marking corruption information providing server 500.

5 is a block diagram illustrating an apparatus 200 for providing information indicating damage to a road surface according to an embodiment of the present invention.

Referring to FIG. 5, the apparatus 200 for providing road surface damage information according to an embodiment of the present invention includes a camera unit 210, a location sensing unit 220, an output unit 230, an image processing unit 240, and a device memory 250, a device storage unit 260, a device communication unit 270, and a device processor 280.

The camera unit 210 may be mounted on a vehicle to photograph a road surface while driving, and output the road surface image to the device processor 280 along with the shooting time. The camera unit 210 may be a luminance camera, and the road surface image may be a video including luminance data and RGB data.

The camera unit 210 is installed to shoot along the driving direction from the road surface of the road closest to the front of the vehicle, as described with reference to FIG. 2, but the vanishing point of the road surface horizontal line in the captured image is 2/3 of the road surface image. It is installed so as to be located within the area to photograph road surface.

The camera unit 210, the location sensing unit 220, the output unit 230, the image processing unit 240, the device memory 250, the device communication unit 270, and the device processor 280 are one device or one module Alternatively, only the camera unit 210 may be implemented as a separate independent device.

The location sensing unit 220 may sense location information of a driving vehicle and output it to the image processing unit 240 or the device processor 280 along with the sensing time.

When it is determined that the road surface marking of the road is damaged or the visibility is poor by the device processor 280, the output unit 230 provides the existence of the road marking in the form of vision or hearing so that the user in the vehicle can recognize it. To make.

The image processing unit 240 may divide a road-matched road surface image into three equal parts from the device processor 280 and set and extract the lower third area as an ROI. The image processing unit 240 uses the extracted ROI as input data of the road surface extraction model.

The device memory 250 may include volatile memory and/or non-volatile memory. The device memory 250 includes, for example, commands or data related to the components 210 to 280 in order to implement and/or provide operations and functions provided by the road surface damaged information providing device 200. The above programs and/or software, operating systems, and the like may be stored.

The program stored in the device memory 250 includes a matching program for visual matching of various sensing data, a road surface extraction model that is learned and generated based on a learning algorithm in the road surface extraction model generation server 100, and damage to the road surface display. It may include a final judgment program for finally determining whether or not.

The device processor 280 executes one or more programs stored in the device memory 250 to control the overall operation of the device 200.

For example, the device processor 280 may execute a matching program to visually synchronize the road surface image input from the camera unit 210 and the location information input from the location sensing unit 220 at the same time. The device processor 280 transmits the time-synchronized road surface image to the image processing unit 240 and inputs and learns the ROI from the image processing unit 240 into the road surface extraction model to temporarily determine whether a road damage area exists in the road surface image. I can judge.

The device processor 280 temporarily determines whether the road marking displayed on the road is damaged based on the visually-synchronized road road image, and finally determines whether the road marking is temporarily damaged or not based on the luminance average of the road marking. can do.

As an example, the device processor 280 extracts a road marking by inputting a road surface image input in real time from the camera unit 210 into a road marking extraction model previously learned based on artificial intelligence, and extracts a road marking. The luminance of the road surface display may be used to finally determine whether or not the road surface display is damaged.

To this end, the device processor 280 may include a road surface display extraction unit 282, a luminance calculation unit 284, and a determination unit 286.

The road surface extracting unit 282 extracts a road surface display area (ie, an outline, (a)) surrounded by the outer surface by applying a road surface extraction model to a road surface image, and the internal damaged area (that is, the inner area) of the extracted road surface display. Line, (b)). If the road marking is not damaged or the degree of damage is less than the reference value, the internal damaged area may not be extracted.

The luminance calculator 284 may calculate the luminance average of the entire road surface display area (a) surrounded by the outline and the luminance average of the internal damaged area (b).

The determination unit 286 may execute a final determination program to determine whether the road surface display is damaged or not, from the difference between the average brightness of the entire road surface and the average of the internal damage area calculated by the brightness calculation unit 284. . Since the final judgment program or the device memory 250 includes a mapping table in which the degree of damage (that is, the degree of poor visibility) is mapped for each difference between the luminance average of the entire road surface display (a) and the luminance average of the internal damage area (b). , The determination unit 286 may check the degree of damage and whether the damage corresponds to the luminance difference calculated from the road surface display included in the road surface image in the mapping table.

For example, if it is determined that the difference between the luminance average and the internal damage area of the entire road surface display belongs to the highest damage class, the determination unit 286 determines the internal damage area as a bad luminance area, and the highest damage Classes can be mapped.

The device processor 280 stores, in the device storage unit 260, the original road road surface image and the road surface image where the road surface image was finally determined, whether the road surface display is damaged or not, and stores a GIS (Geographic Information System). It can also be used for visual service. For example, the device processor 280 may provide a visually differentiated service by checking and applying the color or mark mapped to the maximum damage level in the mapping table when the difference in luminance average corresponds to the maximum damage level. .

The device communication unit 270 may transmit the original image including the location information, the damaged road surface display (that is, the defective luminance area), and the degree of damage to the service providing server 300, as the final determined location of the damaged road surface.

The service providing server 300 may be the server 100 of FIG. 1 described above, and as a separate server, a display device (not shown) with reference to location information for information on a damaged road surface received from the device communication unit 270. ) On the map. The service providing server 300 displays the damaged road surface display area on the map, displays the number of occurrences of each point, the area of occurrence, and the size for each unit section, and uses various statistical processing methods such as cumulative total, average value, standard deviation, and variance. To express the damage characteristics of the unit section. The unit section is a section in which roads are classified by a manager based on predetermined criteria.

The service providing server 300 generates and provides road marking damage information related to the state of the road within a unit section with reference to variables such as road marking damage information, location, occurrence frequency, and size. The road surface damage information for each section can express various damage levels such as 3 or 4 steps, for example, when the degree of damage is serious, the size of the damaged area is large, and the color visually alerts people. The degree of damage in the unit section is expressed by color and quantitative values.

The server 300 may provide information about the damaged road marking, that is, the degree of damage to the road marking and the original road road image and the location information on which the road road image was taken through the GIS.

6 is a diagram illustrating a system for providing damage to road surface information according to another embodiment of the present invention.

Referring to FIG. 6, the road marking damage information system includes an information collecting device 400 and a road marking damage information providing server 500.

The information collecting device 400 is a vehicle-mounted device including a camera 410 and a GPS module 430. The operation of the camera 410 and the GPS module 430 is the same as the camera unit 210 and the location sensing unit 220 described with reference to FIG. 5. However, the information collecting device 400 is mounted on a vehicle and senses only road surface image and location information and transmits it to the road surface damaged information providing server 500.

The road surface display damage information providing server 500 includes a server communication unit 510, a database 520, a server image processing unit 530, a server memory 540, a server processor 550, a server storage unit 560, and a display unit 570 ). The operation of the server image processing unit 530, the server memory 540, the server processor 550, the server storage unit 560, and the display unit 570, the image processing unit 240 described with reference to FIG. 5, the device memory 250 , Since the device processor 280, the device storage unit 260, and the output unit 230 are almost the same, a detailed description thereof will be omitted.

The server communication unit 510 communicates with a plurality of information collection devices 400.

The database 520 stores road surface images and location information received from a plurality of information collection devices 400 together with sensed time information.

The server processor 550 synchronizes the frame and location information of the road surface images received from the information collecting device 400 and then temporarily determines whether there is a damaged road surface display area in the road surface image, and averages the luminance value of the road surface display area. Based on this, it is possible to finally determine whether or not the road marking is damaged or not.

7 is a flowchart illustrating a method of generating a road surface extraction model in the road surface extraction extraction model generation server 100 according to an embodiment of the present invention.

Since the server 100 for the method for generating a road surface extraction model in FIG. 7 has been described with reference to FIGS. 1 to 4, a detailed description thereof will be omitted.

Referring to FIG. 7, the road surface extraction model generation server 100 collects and stores road surface images photographed by a plurality of cameras (S710 ).

The road surface extraction model generation server 100 pre-processes the collected road surface images, converts them into gray images including luminance images, calculates luminance values for each pixel in the gray images, and sets a region of interest (ROI) ( S720).

The road surface extraction model generation server 100 labels and stores the ROI based on the type of road surface display, the luminance of the damaged area, and the damaged size among the ROIs set in step S720 (S730). In step S730, the road marking extraction model generation server 100 extracts the outer contour and the inner contour of the road marking area from among the ROIs, and generates the ROI based on the size of the extracted inner and outer contours and the size and shape of the damaged area predicted by the inner contour. Is labeled with a predetermined color or thickness, and the type of road marking can be mapped and stored in the DB.

The road surface extraction model generation server 100 performs a learning algorithm on a plurality of ROIs labeled by a user to generate a road surface extraction model (S740).

8 is a flowchart illustrating a method for providing road damage information of the road surface damaged information providing apparatus 200 according to an embodiment of the present invention.

Since the apparatus 200 for executing the method for providing road damage information in FIG. 8 has been described with reference to FIG. 5, a detailed description thereof will be omitted.

Referring to FIG. 8, the apparatus 200 for providing road surface damage information photographs a road surface of a driving road using the camera unit 210 and stores location information obtained with a road surface image together with sensed time information. (S810).

The road surface damaged information providing apparatus 200 synchronizes the road surface image stored in step S810 with the location information, and then converts the road surface image including luminance data, that is, an RGB image into a gray image (S820).

The apparatus 200 for providing road surface damage information calculates a luminance value for each pixel in a gray image including luminance data, and then sets an ROI (S830).

The road surface damaged information providing apparatus 200 may apply a road surface extraction model to a set ROI to extract a road surface area surrounded by an outer area from the ROI, and extract an internal damaged area of the extracted road surface display (S840, S850). . In step S850, if the road marking is not damaged or the degree of damage is negligible, the internal damaged area may not be extracted.

The apparatus 200 for providing road surface damage information may calculate a luminance average of the entire road surface display area surrounded by an outer edge and a luminance average of the internal damage area (S860 ).

The road surface display damage information providing apparatus 200 determines whether or not the road surface display is damaged and the degree of damage from the difference between the average brightness of the entire road surface display area and the average of the internal damage area calculated in step S860, and determines the determination result and location information. Together, it may be stored in the device storage unit 260 (S870).

9 is a diagram illustrating an architecture for road surface damage information according to an embodiment of the present invention described above with reference to FIGS. 1 to 8.

Referring to FIG. 9, according to an embodiment of the present invention, a road surface image photographing a road is converted into a gray image including luminance data. According to an embodiment of the present invention, when an ROI area is set and cropped in a gray image, a contouring data set is generated by extracting and labeling the contours (outer and inner lines) of the road surface included in the ROI area. According to an embodiment of the present invention, an AI inference model capable of extracting a road marking from a road surface image and determining whether the road marking is damaged, or poor visibility, by combining the labeling data set and the AI design model, that is, the road marking extraction model Is created as an inference model.

Thereafter, an embodiment of the present invention extracts a gray image including luminance data from an image of a road surface actually photographed, and inputs it to a road surface extraction model to detect a defective luminance region. The defective luminance region may be determined from a difference between a luminance average value of pixels located within the outline of the road surface display and an average luminance value of pixels positioned within the outline of the road surface image ROI.

According to an embodiment of the present invention, the visibility of the bad section of the road surface is visualized by displaying the location of the road determined to have the defective luminance area on the map using the GIS.

On the other hand, an apparatus and method for providing road surface damage information using an image according to the present invention may be provided by being included in a recording medium that can be read through a computer by tangibly implementing a program of instructions for realizing it. It is easily understood by a person skilled in the art.

That is, in order to implement the apparatus and method for providing road surface damage information using the image according to the present invention, a program stored in a computer-readable recording medium performed by the apparatus is provided together.

On the other hand, in the above described and illustrated in connection with a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, deviating from the scope of the technical idea It will be understood by those skilled in the art that many changes and modifications are possible to the present invention. Accordingly, all such suitable changes and modifications and equivalents should also be considered within the scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: road surface extraction model generation server
200: a device for providing information indicating damage to the road surface
300: service providing server
400: information collecting device
500: server for providing road marking damage information

Claims (10)

A device memory in which a damage level is mapped for each difference between a luminance average of the entire road surface display and a luminance average of the internal damage area, and a mapping table in which at least one of colors and marks is mapped for each damage level is stored;
A camera unit mounted on a vehicle and photographing a road being driven to output a road surface image;
The road surface image input from the camera unit is converted into a gray image including luminance information, an area of interest including a road surface display is set in the converted gray image, and the set region of interest is input data of the road surface extraction model As an image processing unit to deliver to the following device processor;
A location sensing unit providing location information of the vehicle; And
The region of interest input in real time from the image processing unit is input to a road surface extraction model previously learned based on artificial intelligence to extract the road surface display displayed on the road, and visibility of the road surface display using the extracted road surface display luminance. It includes; a device processor for determining whether or not damage and the degree of damage, and providing a road surface image in which the road marking is damaged and the degree of damage is determined, and location information on which the road surface image was taken;
The image processing unit,
After equally dividing the road surface image input from the camera unit into a horizontal line and extracting an analysis area corresponding to the lower 1/3 area, an area corresponding to a predetermined position among the analysis areas is set as the area of interest,
The device processor,
A road surface display extraction unit for extracting a road surface display area surrounded by an outer edge by applying the road surface extraction model to a region of interest set in the road surface image, and extracting an internal damaged area of the extracted road surface display area;
A luminance calculator configured to calculate a luminance average of the entire road surface area surrounded by the outer edge and a luminance average of the internal damaged region; And
The difference between the luminance average of the entire road surface display area and the luminance average of the internal damage area calculated by the luminance calculation unit is calculated, and the damage level mapped to the calculated difference is checked from a mapping table stored in the device memory to determine the road surface. Apparatus for providing information indicating damage to the road surface using a video, characterized in that it comprises; a determination unit for determining whether the display is damaged and the degree of damage. delete delete delete According to claim 1,
The camera unit,
It is installed to shoot along the driving direction from the road surface of the road closest to the front surface of the vehicle, the vanishing point of the road surface horizontal line in the captured image is installed so that the image characterized in that located in the 2/3 area of the road surface image A device for providing damage to road markings on roads used. (A) the electronic device, photographing the road with a camera mounted on the vehicle to output the road surface image;
(B) the electronic device sensing the location information of the vehicle using a GPS module; And
(C) the electronic device converts the road surface image output in step (A) into a gray image including luminance information, sets an area of interest including a road surface display in the converted gray image, and sets the Outputting a region of interest as input data of a road surface extraction model;
(D) The electronic device inputs a region of interest input in real time in step (C) into a road marking extraction model previously learned based on artificial intelligence, extracts a road marking displayed on the road, and extracts the road marking. Determining whether the visibility of the road surface is damaged and the degree of damage by using the luminance of the display, and providing a road surface image in which the road surface image is damaged and the degree of damage, and location information on which the road surface image is taken;
Including,
Step (C) is,
After equally dividing the road surface image input from the camera into horizontal lines and extracting an analysis area corresponding to the lower 1/3 area, setting an area of interest corresponding to a predetermined position among the analysis areas,
Step (D) is,
(D1) the electronic device applying the road surface extraction model to a region of interest set in the road surface image, extracting a road surface display area surrounded by an outline, and extracting an internal damaged area of the extracted road surface display area;
(D2) calculating, by the electronic device, a luminance average of the entire road surface area surrounded by the outer edge and a luminance average of the internal damaged region; And
(D3) The electronic device calculates a difference between a luminance average of the entire calculated road surface display area and a luminance average of the internal damage area, and confirms the damage level mapped to the calculated difference from a mapping table to display the road surface Determining whether the damage and the degree of damage; Method of providing information on the road surface damage information using the image of the electronic device comprising a. delete delete delete The method of claim 6,
The camera,
An electronic device installed to photograph along the driving direction from the road surface of the road closest to the front surface of the vehicle, wherein the vanishing point of the horizontal road surface in the captured image is located within 2/3 of the road surface image. How to provide road marking damage information using the video.

Images