KR102466004B1 - Lane Painting System using Deep Learning Technology - Google Patents

Abstract

The present invention relates to a lane painting system using deep learning technology, which can precisely and accurately detect existing lanes without being affected by environmental factors by using deep learning technology, and can automatically control a lane painting vehicle and a nozzle so that the straight lines and curves of existing lanes can be distinguished and repainted smoothly. To solve the above problems, the lane painting system using deep learning technology comprises: a lane photographing unit for photographing the lanes of the road in real time; a preprocessing unit for increasing the lane recognition rate by processing images captured by the lane photographing unit; a learning model generation unit for generating a lane learning model by iteratively learning lane types of image data preprocessed by the preprocessing unit; a painting vehicle and nozzle control learning unit for learning lane painting control values through painting vehicle control data for each lane situation, nozzle control data, and correction data resulting from painting results; and a painting vehicle and nozzle control unit for recognizing lanes according to real-time lane images based on the lane learning model generated by the learning model generation unit and the lane painting control values learned by the painting vehicle and nozzle control learning unit, extracting the lane painting control values resulting from recognized lane situations, and controlling the lane painting vehicle and the nozzle.

Description

Lane Painting System using Deep Learning Technology}

The present invention relates to a lane painting system using deep learning technology, and more particularly, deep learning that automatically controls the driving of a painted vehicle and nozzles for spraying paint in order to accurately repaint the lane along the existing lanes painted on the road. It is about a lane painting system using technology.

In general, lanes are painted on roads to guide the driving direction and spacing of vehicles. As these lanes are exposed to the outside for a long time, visibility deteriorates, making it difficult to maintain their original functions. This is to ensure that the original function is maintained.

As a prior art for the above purpose, an automatic lane painting device using a vision sensor of Patent Registration No. 2143641 (hereinafter referred to as 'Patent Document') is disclosed.

The lane painting device disclosed in the patent document includes a front camera installed at the front side of a vehicle and acquiring a real-time image by photographing the lane; a lane painting unit located at the lower part or one side of the vehicle and painting the lane according to the position of the lane; a controller installed in the vehicle and receiving real-time images acquired through the front camera and then converting them into data to control the operation of the lane painting unit; and a display unit for outputting a real-time image captured by the front camera, wherein the lane painting unit includes a rear camera that recognizes the lane and obtains location information, and the controller includes a driving direction of the vehicle and A plurality of sensors for detecting the lane are installed, and a driving direction of the vehicle and a recognition rate of the lane are corrected through the plurality of sensors, and the plurality of sensors sense the driving direction of the vehicle and simultaneously recognize the lane IMU sensor for detecting the position of; an illuminance sensor for recognizing the lane on the road; and a multi-spectrum sensor recognizing a specific material of the lane painted on the road using a wavelength of a selected range, wherein the control unit merges information of the lane recognized through the illuminance sensor and the multi-spectrum sensor. to correct the information of the damaged lane, and apply a predetermined angle so that the image of the road and the lane photographed through the front camera is perpendicular to the road based on the angle at which the front camera is installed. A bird's-eye view image is generated, a binarization algorithm is applied to the image detected by the rear camera to divide the road area and the lane area, extract feature points of the divided lane area, and then track the line based on the extracted feature points. The damaged part of the lane is compensated for, and the rear camera is spaced apart from the rear of the front camera by a predetermined distance and is installed to photograph the lane in a direction perpendicular to the road, and the control unit takes pictures through the front camera. A virtual lane guide is merged with the generated image and output through the display unit, and the lane guide has one end of a first side bar formed to have a predetermined straight length along the lane on one side of the first side bar. A connection bar having a predetermined length so that the other end is orthogonal to the first side bar while being connected; And a second side bar having one end connected to the other end of the connecting bar and having a predetermined straight length such that the other end is parallel to the first side bar, and the width between the first side bar and the second side bar is is formed with a predetermined width wider than the width of, a left separation distance is formed between the first side bar and the left side of the lane, and a right separation distance is formed between the second side bar and the right side of the lane, and the control unit, If the left and right separation distances are the same, it is determined that the vehicle is traveling accurately along the lane, and if the left and right separation distances are different from each other, it is determined that the vehicle is driving in a direction with a relatively small value, It consists in controlling the running direction of the vehicle so that the right separation distance is kept equal to each other.

The lane painting device disclosed in the patent document detects the position and width of an existing vehicle based on information detected through a camera and a sensor, and based on this, controls the driving direction of the vehicle to automatically paint the lane, but in this case, real-time It is difficult to quickly control the driving direction of the vehicle according to the captured images and detection signals, and if a delay occurs in controlling the driving direction of the vehicle, a positional error occurs between the position of the existing lane and the newly painted lane, and the painted vehicle Even if you drive accurately along the lane, the direction of the vehicle and the direction of the nozzle move in conjunction with each other, so in the case of a curved lane like a curve, the distance between the vehicle and the existing lane changes and the position of the nozzle easily deviate from the position of the existing lane. There is a problem that the accuracy of repainting of curved lanes is lowered due to

Therefore, it is required to develop an improved lane painting system capable of accurately detecting lanes for automation of lane painting and repainting lanes accurately by controlling a painting vehicle and nozzles along the detected lanes.

KR 10-2143641 B1 (2020. 08. 05.) KR 10-2021-0071724 A (2021. 06. 16.) KR 10-2257533 B1 (2021. 05. 24.) KR 10-2078908 B1 (2020. 02. 12.) KR 10-2019-0054656 A (2019. 05. 22.)

The present invention was made to solve the problems of the conventional automatic lane painting apparatus as described above, and the problem to be solved by the present invention is to accurately and accurately paint existing lanes without being affected by environmental factors by using deep learning technology. To provide a lane painting system using deep learning technology that can detect and automatically control the lane painting vehicle and nozzle to distinguish straight lines and curves of existing lanes and smoothly repaint them.

A lane painting system using deep learning technology according to the present invention for solving the above problems includes a lane photographing unit for photographing a lane of a road in real time; a pre-processing unit to increase a lane recognition rate by processing the image captured by the lane capturing unit; a learning model generation unit that generates a next best learning model by iteratively learning the next best type of the image data preprocessed by the preprocessing unit; a painting vehicle and nozzle control learning unit that learns a lane painting control value through painting vehicle control data for each lane situation, nozzle control data, and correction data according to a painting result; and based on the lane learning model generated by the learning model generation unit and the lane painting control value learned by the painting vehicle and nozzle control learning unit, the lane is recognized according to the real-time lane image, and the lane is painted according to the situation of the recognized lane. It is characterized in that it includes a painting vehicle and nozzle controller for controlling the painting vehicle and the nozzle by extracting a control value.

And, the present invention includes first, second, and third cameras arranged in a row by the lane photographing unit, and the first camera is installed perpendicular to the road so that the lane at a position close to the nozzle is photographed without perspective, , The second camera is installed so that a part of the lane photographed by the first camera and a part of the lane in front of the nozzle are included and photographed, and the third camera has a predetermined length photographed by the second camera. Another feature is that the lane and the lane of a predetermined length toward the front of the shooting area of the second camera are included and photographed.

In addition, the present invention includes a resolution adjusting unit for adjusting the image data resolution of the image taken by the lane photographing unit in the pre-processing unit; a data conversion unit for converting color data of the image data adjusted by the resolution adjustment unit; a noise removal unit which removes noise by performing Gaussian processing on the image data obtained by converting the color data in the data conversion unit; a binarization processing unit which binarizes the image data from which the noise has been removed through the noise removal unit; and a perspective conversion unit rearranging and removing perspective elements of the image data binarized by the binarization processing unit.

In addition, in the present invention, the learning model generation unit recognizes lanes from the image data preprocessed in the preprocessing unit, sets control values of the painting vehicle and nozzles according to the twist and curvature of the recognized lane, and sets the painting vehicle and nozzles. Another feature is to generate a suboptimal learning model including a control value of .

And in the present invention, the control value of the painting vehicle and the nozzle includes the position of the nozzle, the height of the nozzle, the spray amount and the spray speed value according to the speed, steering, center line of the lane, width of the lane, twist and curvature of the paint vehicle has another feature.

According to the present invention, lanes can be accurately detected through repetitive learning by artificial intelligence deep learning technology even under the influence of sunlight, shadows, and lane defects, and through this, it is possible to accurately control a painting vehicle and nozzles for repainting lanes. There are advantages to being able to

In addition, it is possible to optimally control the painting vehicle and nozzle by distinguishing the straight line and the curve of the lane, and through this, it is possible to reliably reduce the re-painting lane deviating from the existing lane. It has the advantage of reducing manpower and cost.

1 is a diagram showing an example of a lane painting system using deep learning technology according to the present invention.
2 to 4 are views showing an example of a lane capturing unit according to the present invention.
5 is a view showing an example of a pre-processing unit according to the present invention.
6A to 6H are diagrams showing examples of pre-processing image data through a pre-processing unit according to the present invention;
7 and 8 are views showing an example of repainting a lane through a painting vehicle and a nozzle control unit according to the present invention.

Hereinafter, it will be described in detail according to the accompanying drawings showing a preferred embodiment of the present invention.

The present invention uses deep learning technology to precisely and accurately detect existing lanes without being affected by environmental factors, and automatically controls the lane painting vehicle and nozzle to distinguish straight lines and curves of existing lanes and smoothly repaint them. It is intended to provide a lane painting system using deep learning technology that can be used, and the present invention for this purpose includes a lane photographing unit 10, a preprocessing unit 20, and a learning model generating unit 30 as shown in FIGS. 1 and 2. ), a painting vehicle and nozzle control learning unit 40 and a painting vehicle and nozzle control unit 50.

The lane photographing unit 10 is installed on one side of the painting vehicle 1 to photograph the lane 3 on the road in real time and transmits the photographed image to a pre-processing unit 20 to be described later.

In the lane photographing unit 10, a plurality of cameras are installed at different angles to photograph different areas Z1, Z2, and Z3 along the lane 3, and through this, in the learning model generation unit 30 to be described later Images of different angles are created as learning models.

More specifically, as shown in FIGS. 2 to 4, it is composed of first, second, and third cameras 11, 12, and 13 arranged in a row with each other, and the first camera 11 is close to the nozzle 2. The lane 3 at the position is installed so that it is perpendicular to the road so that it is photographed without perspective, and the second camera 12 is a certain length of the front side of the nozzle 2 and some of the lanes photographed by the first camera 11 The third camera 13 includes a part of the lane captured by the second camera 12 and a lane of a predetermined length toward the front of the shooting area of the second camera 12. It can be installed to be filmed.

At this time, the first, second, and third cameras 11, 12, and 13 are configured to be able to adjust the angle, and the installation angle of the first, second, and third cameras 11, 12, and 13 will be described later as needed for a painting vehicle. And it may be configured to be automatically adjusted by the control of the nozzle control unit 50.

To this end, a rotation shaft (not shown) having a predetermined length is installed in the first, second, and third cameras 11, 12, and 13, and a power transmission member (not shown) such as a gear or a pulley is installed on the rotation shaft, Rotational force may be transmitted through the power transmission member and a rotational power generating device (not shown) such as a motor.

In addition, the first, second, and third cameras 11, 12, and 13 may be configured to adjust the height of the distance from the road. For this purpose, the first, second, and third cameras 11, 12, and 13 are vertically After being installed on a linear guide (not shown) having a predetermined length, the height may be adjusted while moving up and down by the operation of a linear motor.

On the other hand, through the first, second, and third cameras 11, 12, and 13, a predetermined width is overlapped and photographed. The images of the three regions Z1, Z2, and Z3 are grouped into one set and subjected to image processing in the pre-processing unit 20 to be described later.

Through this, images of various angles according to the condition of the same lane, such as color, shape, angle, and defect, can be simultaneously acquired and created as a learning model. Even if it is, it is possible to quickly extract past data of similar conditions from the learning model and obtain the optimal control value (painting vehicle, nozzle) suitable for it.

The pre-processing unit 20 is a component that increases the recognition rate of lanes by processing images captured by the lane capturing unit 10 .

As shown in FIG. 5, the pre-processing unit 20 includes a resolution adjusting unit 21 for adjusting the resolution of image data of an image captured by the lane capturing unit 10, and image data adjusted by the resolution adjusting unit 21. A data conversion unit 22 for converting color data, a noise removal unit 23 for removing noise by performing Gaussian processing on the image data converted from color data in the data conversion unit 22, and the noise removal unit 23 ) and a binarization processing unit 24 that binarizes the image data from which noise has been removed through and a perspective conversion unit 25 that rearranges and removes perspective elements of the image data binarized by the binarization processing unit 24.

As an example of such a pre-processing unit 20, as shown in FIGS. 6A and 6B, the image taken by the lane capturing unit 10 is converted into an image of 640x360 resolution through the resolution adjusting unit 21, and then the data conversion unit Through (22), the color space is changed so that the lane detection is easy. At this time, in general, since white uses RGB color space and yellow uses HSV color space, the data converter 22 changes to RGB color space so that the white line can be easily detected.

Thereafter, as shown in FIG. 6C through the noise removal unit 23, a specific color in the image is color-masked and then other colors are removed, and as shown in FIG. 6D, Gaussian Blur is applied.

At this time, it may be difficult to extract an accurate lane boundary because the boundary of the lane becomes dull according to the applied equator of the Gaussian blur. Therefore, a value previously set by the user may be applied as the applied value of the Gaussian blur.

Thereafter, as shown in FIG. 6E, white noise in the image data is removed through morphological calculation, and then black noise in the white object is removed through additional morphological calculation as shown in FIG. 6F.

And, as shown in FIG. 6G through the binarization processing unit 24, in order to clarify the boundary in the image data, it is binarized so that there is no intermediate color between black and white.

Then, through the perspective conversion unit 25, as shown in FIG. 6H, the perspective value of the lane in the binarized image data is adjusted, and the effective region of the lane is designated.

At this time, if the effective area of the lane is wide, confusion with other lanes or errors due to perspective adjustment may occur. Therefore, it is preferable to designate a lane with a length of 2 to 3M based on the nozzle 2 (origin) as the effective area.

The preprocessed image data is transmitted to the learning model generation unit 30 to be described later.

Through the configuration as described above, the effect of shadows, sunlight, and defects (contamination of lanes, loss of paint of lanes, cracks in roads, damage to roads, etc.) included in the image captured by the lane photographing unit 10 The false recognition rate can be reduced.

The learning model generating unit 30 is a component that repeatedly learns the next best type of the image data preprocessed by the preprocessing unit 20 to generate a next best learning model.

The learning model generation unit 30 recognizes lanes from the image data preprocessed by the preprocessing unit 20, sets the control values of the painting vehicle and nozzles according to the twist and curvature of the recognized lane, and sets the painting vehicle and A second best learning model is created including the control value of the nozzle.

At this time, the control values of the painting vehicle and the nozzle may include the location of the nozzle, the height of the nozzle, the amount of injection, and the value of the injection speed according to the speed of the painting vehicle, steering, center line of the lane, width of the lane, twist and curvature of the lane.

Hereinafter, an example of recognizing a lane in the learning model generation unit 30 according to the present invention will be described.

As shown in FIG. 7, the outline of a figure (lane) is extracted from the preprocessed image data and the boundary is connected.

Then, a virtual left lane line IL and a virtual right lane line IR are generated by connecting the left and right outermost lines among the boundary lines of the outline.

Further, the widths W1 and W2 of the upper and lower lanes connecting the imaginary left lane line IL and the imaginary right lane line IR are measured, and the widths W1 and W2 of the upper and lower lanes are the same as each other. Spacing is adjusted so that it is wide.

Thereafter, a center line CL of the lane connecting the 1/2 point of the widths W1 and W2 of the upper and lower lanes is created, and the position of the nozzle 2 is followed along the center line CL of the lane. The nozzle position value is set.

After the nozzle position value is set as described above, the paint vehicle control data and nozzle control data for each lane condition are automatically set according to the preset road regulations.

The painting vehicle and nozzle control learning unit 40 generates correction data of past lane repainting results through the painting vehicle control data and nozzle control data for each lane situation, and the painting vehicle control data for each lane situation, nozzle control data and It is a configuration that repeatedly learns the correction data to generate an optimal lane painting control value that does not cause errors such as lane departure due to lane width and twist, which occur when the lane is repainted along the center line (CL) of the lane.

The painting vehicle and nozzle control unit 50 generates a lane according to the real-time lane image based on the lane learning model generated by the learning model generator 30 and the lane painting control value learned by the painting vehicle and nozzle control learning unit 40. It is a configuration that controls the paint vehicle and nozzle by recognizing and extracting the lane painting control value according to the recognized lane situation.

At this time, based on the lane painting control value generated by the painting vehicle and nozzle control unit 50, a predetermined error value M is a virtual left lane line IL and a virtual right lane based on the center line CL of the lane. It is applied separately to the line IR, so that the repainting of the lane proceeds.

As described above, the present invention can accurately detect lanes through repetitive learning by artificial intelligence deep learning technology even under the influence of sunlight, shadows, and lane defects, and through this, paint vehicles and nozzles for lane repainting can be accurately be able to control

In addition, it is possible to optimally control the painting vehicle and nozzle by distinguishing the straight line and the curve of the lane, and through this, it is possible to reduce the re-painting lane deviating from the existing lane. will be able to reduce costs.

In the above, reference numerals and names have been given to the drawings showing preferred embodiments and the components shown in the drawings for convenience of explanation, but this is an embodiment according to the present invention, which corresponds to the shape shown on the drawings and the given name. The scope of the right should not be construed as being limited, and the change to various shapes predictable from the description of the invention and the simple substitution with a configuration that has the same action are within the range of change to be easily performed by those skilled in the art. You will find it extremely self-evident.

1: Paint vehicle 2: Nozzle
3: lane 10: lane shooting department
11: first camera 12: second camera
13: third camera 20: pre-processing unit
21: resolution adjustment unit 22: data conversion unit
23: noise removal unit 24: binarization processing unit
25: perspective conversion unit 30: learning model generation unit
40: painting vehicle and nozzle control learning unit 50: painting vehicle and nozzle control unit
CL: the center line of the lane IL: the imaginary left lane line
IR: imaginary right lane line M: error value
W: the width of the lane W': 1/2 of the lane width
W1: Width of the upper lane W2: Width of the lower lane

Claims (5)

Lane photographing unit 10 for photographing lanes of the road in real time;
a pre-processing unit 20 to increase a lane recognition rate by processing the image captured by the lane capturing unit 10;
a learning model generation unit 30 that generates a next best learning model by iteratively learning the next best type of the image data preprocessed by the preprocessing unit 20;
a painting vehicle and nozzle control learning unit 40 that learns a lane painting control value through the painting vehicle control data for each lane situation, the nozzle control data, and the correction data according to the painting result; and
Based on the lane learning model generated by the learning model generation unit 30 and the lane painting control values learned by the painting vehicle and nozzle control learning unit 40, the lane is recognized according to the real-time lane image, and the recognized lane a paint vehicle and nozzle controller 50 that controls the paint vehicle and the nozzle by extracting a lane painting control value according to circumstances;
including,
The lane photographing unit 10,
Including first, second, and third cameras 11, 12, and 13 arranged in one row with each other,
The first camera 11,
It is installed to be perpendicular to the road so that a lane in a position close to the nozzle is photographed without perspective,
The second camera 12,
It is installed so that a part of the lane photographed by the first camera 11 and a lane of a certain length in front of the nozzle are included and photographed,
The third camera 13,
It is installed so that a part of the lane photographed by the second camera 12 and a lane of a predetermined length in front of the photographing area of the second camera 12 are included and photographed,
The learning model generator 30,
The lane is recognized from the image data preprocessed by the pre-processing unit 20, the paint vehicle and nozzle control values are set according to the twist and curvature of the recognized lane, and lane learning is included including the set paint vehicle and nozzle control values. Lane painting system using deep learning technology, characterized in that for generating a model.
delete The method of claim 1,
The pre-processing unit 20,
a resolution adjusting unit 21 for adjusting the resolution of image data of the image captured by the lane capturing unit 10;
a data conversion unit 22 for converting color data of the image data adjusted by the resolution adjustment unit 21;
a noise removal unit 23 which removes noise by performing Gaussian processing on the image data whose color data has been converted by the data conversion unit 22;
a binarization processing unit 24 for binarizing the image data from which noise has been removed by the noise removal unit 23; and
a perspective conversion unit 25 rearranging and removing perspective elements of the image data binarized by the binarization processing unit 24;
Lane painting system using deep learning technology, characterized in that it comprises a.
delete The method of claim 1,
The control values of the painting vehicle and the nozzle are
Lane painting system using deep learning technology, characterized in that it includes the nozzle position, nozzle height, spray amount and spray speed values according to the speed, steering, center line of the lane, width of the lane, twist and curvature of the vehicle to be painted.

Images