KR102540636B1 - Method for create map included direction information and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

A method for generating a map including directional information for traffic facilities existing on a road is proposed. The method comprises: a step in which a map generation unit receives at least one image captured by at least one camera fixed on a vehicle; a step in which the map generation unit detects traffic facilities in the received at least one image; a step in which the map generation unit generates directional information based on point cloud data acquired by a lidar corresponding to the detected traffic facilities; and a step in which the map generation unit generates a map including the generated directional information. The present invention is a technology developed through the Urban Change Detection Platform Using Mobility Services at the 2022 Seoul Innovation Challenge (Finals) (IC210027) conducted by the Seoul Industry Promotion Agency. Additionally, The present invention is a technology developed through Project Number 22AMDP-C160637-02 by the National Institute of Land, Infrastructure and Transport Science and Technology under the Ministry of Land, Infrastructure and Transport. Therefore, the method allows effective use of precision road maps through the generated directional information.

Description

Method for creating map included direction information and computer program recorded on record-medium for executing method therefor}

The present invention relates to maps with precision. More specifically, it relates to a method for generating a map including direction information on traffic facilities existing on a road and a computer program recorded on a recording medium to execute the method.

Autonomous driving of a vehicle refers to a system that can judge and drive a vehicle by itself. Such autonomous driving may be classified into gradual stages from non-automation to complete automation according to the degree of involvement of the system in driving and the degree of control of the vehicle by the driver. In general, the level of autonomous driving is divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the six levels classified by the International Association of Automotive Engineers, level 0 is non-automation, level 1 is driver assistance, level 2 is partial automation, level 3 is conditional automation, level 4 is highly automated, and level 5 The steps are fully automated steps.

Autonomous driving is performed through mechanisms of perception, localization, path planning, and control. In addition, various companies are developing to implement recognition and path planning in autonomous driving mechanisms using artificial intelligence (AI).

For such autonomous driving, various information on roads must be preemptively collected. However, in reality, it is not easy to collect and analyze vast amounts of information in real time using only vehicle sensors. Accordingly, in order for autonomous driving to become a reality, maps with high precision that can provide various information necessary for actual autonomous driving are essential.

Here, the high-precision road map refers to a three-dimensional electronic map constructed with information on roads and surrounding topography with an accuracy of ±25 cm. In addition to general electronic map information (node information and link information necessary for route guidance), these precision maps include road width, road curvature, road slope, lane information (dotted lines, solid lines, stop lines, etc.) It is a map that includes precise information such as surface type information (crosswalks, speed bumps, shoulders, etc.), road surface mark information, sign information, facility information (traffic lights, curbs, manholes, etc.).

In order to create a map with such precision, various related data such as a Mobile Mapping System (MMS) and aerial photographic information are required.

In particular, the MMS is mounted on a vehicle to measure the position of a feature in the vicinity of a road and acquire visual information while driving the vehicle. That is, the MMS collects GPS (Global Positioning System), Inertial Navigation System (INS), Inertial Measurement Unit (IMU), shape and information of landmarks to acquire position and attitude information of the vehicle body. It may be generated based on information collected by a camera, a LiDAR (Light Detection and Ranging) sensor, and other sensors for processing.

These high-precision maps produced using MMS include various traffic facilities such as various signs and traffic lights. Accordingly, in order to effectively use the map with high precision, direction information on traffic facilities is required.

Recently, algorithms for generating direction information by using point cloud data obtained from LIDAR have been developed, but their structure is very complicated and their usability is poor.

In addition, in order to generate learning data about traffic facilities included in the map with high precision, a large amount of images including traffic facilities are required. However, it was difficult to obtain data because there is no publicly available data set for images including traffic facilities and the amount of images including traffic facilities is limited.

On the other hand, when 3D modeling is performed using an aerial lidar, since point cloud data obtained from a lidar located on the upper part of an object is collected, the density of the point cloud collected decreases toward the lower part of the object. Accordingly, since the existing modeling techniques perform modeling on all the collected point clouds at once, noise is generated according to the difference in density of the point clouds, and as a result, there is a problem in that the accuracy of the modeled object is lowered.

Meanwhile, in general, in order to generate a color map by assigning a color of an image captured from a camera to point cloud data obtained from a lidar, point cloud data accumulated in units of a specific distance is projected onto a single image. Accordingly, the generated color map is expressed as being occluded by a specific object on the image, but there is a problem in that noise is generated in which the wrong color of the image is applied to the recognized area on the point cloud data.

In addition, the generated color map has a problem in that noise is generated in which a bright color is applied to an object having a specific dark color due to the color of the sky or light reflection.

The present invention is a technology developed through the Seoul Business Agency (2022 Seoul Innovation Challenge (final)) (IC210027) 'Urban change detection platform using mobility service'.

In addition, the present invention is a technology developed through 'Task No. 22AMDP-C160637-02' of the Ministry of Land, Infrastructure and Transport under the Ministry of Land, Infrastructure and Transport Science and Technology Promotion Agency.

Republic of Korea Patent Publication No. 10-2013-0123041, 'Method of automatically generating indoor map using LIDAR device', (published on November 12, 2013)

One object of the present invention is to provide a method for generating a map including direction information about traffic facilities existing on a road.

Another object of the present invention is to provide a computer program recorded on a recording medium in order to execute a method of generating a map including direction information about traffic facilities existing on a road.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, the present invention proposes a map generation method including direction information about traffic facilities existing on a road. The method includes receiving, by a map generator, at least one image captured by at least one camera fixed to a vehicle, and detecting, by the map generator, a traffic facility on the received at least one image. , generating, by the map generating unit, direction information based on point cloud data obtained by a lidar corresponding to the detected traffic facility, and generating a map including the generated direction information by the map generating unit. It includes the steps of creating

Specifically, the detecting of the traffic facility may include detecting the traffic facility on the 2D image using a deep learning-based object detection algorithm.

The generating of the direction information may include generating direction information when the detected traffic facility is located within a preset distance.

The generating of the direction information may include generating direction information when the number of point groups included in the detected traffic facility is equal to or greater than a preset value.

The generating of the direction information may include generating the direction information when a ratio between a horizontal width and a vertical width of a bounding box of the detected traffic facility is equal to or less than a preset value.

The detecting of traffic facilities may include detecting traffic facilities on the 2D image using distance-based Euclidean clustering.

The generating of the direction information may include filtering and down-sampling the point cloud included in the detected traffic facility in the form of a voxel grid.

The generating of the direction information may include obtaining a representative plane by randomly searching the point cloud for the detected traffic facility using a random sample consensus (RANSAC) algorithm.

The generating of the direction information may include acquiring the representative plane when a predetermined value is satisfied by accumulating point cloud data for the detected traffic facility.

The generating of the direction information may include obtaining the representative plane by accumulating the point cloud data for the detected traffic facility by a predetermined number of revolutions of the lidar.

The generating of the direction information may include calculating a normal vector extending vertically from the plane using a plane equation based on the representative plane.

The generating of the map may include including the generated direction information in meta information of the map.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute a method of generating a map including direction information about traffic facilities existing on a road. The computer program may be combined with a computing device comprising a memory, a transceiver, and a processor that processes instructions resident in the memory. The computer program may include receiving, by the processor, at least one image taken by at least one camera fixed to a vehicle; detecting, by the processor, a traffic facility on the received at least one image; generating, by the processor, direction information based on point group data obtained by a lidar corresponding to the detected traffic facility; and by the processor, the generated direction information is It may be a computer program recorded on a recording medium to execute the step of generating the included map.

Details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, by generating direction information for traffic facilities detected on an image based on point cloud data, it is possible to effectively use a map with precision through the generated direction information.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a configuration diagram showing a data generating system according to an embodiment of the present invention.
2 is a logical configuration diagram of a data generating device according to an embodiment of the present invention.
3 is a hardware configuration diagram of a data generating device according to an embodiment of the present invention.
4 is a flowchart illustrating a method of generating a map including direction information according to an embodiment of the present invention.
5 is a flowchart illustrating a method of generating learning data for a traffic facility according to an embodiment of the present invention.
6 is a flowchart illustrating a method of generating a map using an aerial lidar according to an embodiment of the present invention.
7 is a flowchart illustrating a projection-based color map generation method according to an embodiment of the present invention.
8 is a flowchart illustrating a method for generating a color map to which color correction is applied according to an embodiment of the present invention.
9 and 10 are exemplary diagrams for explaining a method of generating a map including direction information according to an embodiment of the present invention.
11 to 13 are exemplary diagrams for explaining a method of generating learning data for traffic facilities according to an embodiment of the present invention.
14 to 16 are exemplary diagrams for explaining a map generation method using an aerial lidar according to an embodiment of the present invention.
17 and 18 are exemplary diagrams for explaining a projection-based color map generation method according to an embodiment of the present invention.
19 and 24 are exemplary diagrams for explaining a color map generation method to which color correction is applied according to an embodiment of the present invention.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are excessively inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as 'consisting of' or 'having' should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as "'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

On the other hand, the high-precision map produced using MMS includes various traffic facilities such as various signs and traffic lights. Accordingly, in order to effectively use the map with high precision, direction information on traffic facilities is required.

Recently, algorithms for generating direction information by using point cloud data obtained from LIDAR have been developed, but their structure is very complicated and their usability is poor.

In addition, in order to generate learning data about traffic facilities included in the map with high precision, a large amount of images including traffic facilities are required. However, it was difficult to obtain data because there is no publicly available data set for images including traffic facilities and the amount of images including traffic facilities is limited.

On the other hand, when 3D modeling is performed using an aerial lidar, since point cloud data obtained from a lidar located on the upper part of an object is collected, the density of the point cloud collected decreases toward the lower part of the object. Accordingly, since the existing modeling techniques perform modeling on all the collected point clouds at once, noise is generated according to the difference in density of the point clouds, and as a result, there is a problem in that the accuracy of the modeled object is lowered.

Meanwhile, in general, in order to generate a color map by assigning a color of an image captured from a camera to point cloud data obtained from a lidar, point cloud data accumulated in units of a specific distance is projected onto a single image. Accordingly, the generated color map is expressed as being occluded by a specific object on the image, but there is a problem in that noise is generated in which the wrong color of the image is applied to the recognized area on the point cloud data.

In addition, the generated color map has a problem in that noise is generated in which a bright color is applied to an object having a specific dark color due to the color of the sky or light reflection.

In order to overcome these limitations, the present invention can generate a map including direction information on traffic facilities present on the road, facilitate recognition of traffic facilities present on the road, and lidar installed in an airborne device. We propose various means that can efficiently create a 3D map using the point cloud data obtained from LIDAR and generate a color map by assigning the color of an image taken from a camera to the point cloud data obtained from LIDAR.

1 is a configuration diagram showing a data generating system according to an embodiment of the present invention.

As shown in FIG. 1 , a data generating system 300 according to an embodiment of the present invention may include a data collecting device 100 and a data generating device 200 .

Since the components of the data generation system according to one embodiment are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is implemented in an actual physical environment. They may be implemented separately from each other.

Each of the components will be described. The data collection device 100 may collect data necessary for generating a map and generating learning data.

The data collection device 100 may include one of a lidar, a camera, a radar, an ultrasonic sensor, an encoder, a temperature sensor, and a humidity sensor. It may be configured to include the above. However, the data collection device 100 is not limited thereto, and sensors capable of sensing various types of information may be applied to the data collection device 100 to generate a precise road map.

In particular, the data collection device 100 according to an embodiment of the present invention may include a lidar and a camera, and may collect point cloud data acquired by the lidar and images captured by the camera.

Here, lidar emits laser pulses around the vehicle 10 or airborne device 20 and detects light reflected by objects located around the vehicle and returns, resulting in a 3D image corresponding to the surroundings of the vehicle. Point cloud data can be created.

The camera may obtain an image of a space collected from the lidar centered on the lidar. Such a camera may include any one of a color camera, a near infrared (NIR) camera, a short wavelength infrared (SWIR) camera, and a long wavelength infrared (LWIR) camera.

The data collection device 100 may be installed in the vehicle 10 or the airborne device 20 . For example, the data collection device 100 is installed on the upper part of the vehicle 10 to collect point cloud data or images of the surroundings, or is installed on the lower part of the airborne device 20 and is installed on the ground object in the air. Data or images may be collected.

With the following configuration, the data generating device 200 may receive point cloud data acquired by the LIDAR and an image captured by the camera from the data collecting device 100 .

The data generating device 200 may generate a detailed road map using the received point cloud data and a camera, and may generate learning data using the detailed road map.

Characteristically, according to an embodiment of the present invention, the data generating device 200 receives at least one image taken by at least one camera fixed to the vehicle 10, and traffic is transmitted on the received at least one image. A facility may be detected, direction information may be generated based on point cloud data obtained by LIDAR for the detected traffic facility, and a map including the generated direction information may be generated.

According to another embodiment of the present invention, the data generating device 200 collects traffic facility sample images, and processes the collected traffic facility sample images to be recognized as images captured by a camera installed in a vehicle. Learning data may be generated by generating an image and inserting a background into at least one generated processed image.

In addition, according to another embodiment of the present invention, the data generating device 200 receives point cloud data acquired from lidar installed in the airborne device 20, and displays a top-view of the received point cloud data. A map of an object can be created by projecting and scanning the projected point cloud data in a direction perpendicular to the ground.

In addition, according to another embodiment of the present invention, the data generating apparatus 200 receives an image captured by a camera and point cloud data obtained from LIDAR, projects the received point cloud data onto an image, and projects the received point cloud data onto the image. A color map can be created by filtering the filtered point cloud data based on distance and assigning an image color to the filtered point cloud data.

In addition, according to another embodiment of the present invention, the data generating apparatus 200 assigns a color of an image captured by a camera to point cloud data obtained from LIDAR, and sets a preset color in the color-added point cloud data. A color map can be generated by separating the points having the same color and correcting the color of the separated points based on the colors of neighboring points.

Meanwhile, various embodiments of the present invention have been described as being separated from each other and performing respective functions, but the present invention is not limited thereto and may be applied in combination with each other.

As described above, the data collecting device 100 and the data generating device 200 transmit and receive data using a network in which at least one of a secure line, a common wired communication network, and a mobile communication network directly connects devices. can do.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

Hereinafter, the data generating device 200 as described above will be described in detail.

2 is a logical configuration diagram of a data generating device according to an embodiment of the present invention.

Referring to FIG. 2 , the data generating device 200 according to an embodiment of the present invention includes a communication unit 205, an input/output unit 210, a storage unit 215, a data collection unit 220, and a map generation unit 225. ) and a learning data generator 230.

Since the components of the data generating device 200 are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is separated from each other in an actual physical environment. and can be implemented.

Describing each component, the communication unit 205 may transmit and receive data to and from the data collection device 100 . Specifically, the communication unit 205 may receive point cloud data obtained by LIDAR and an image captured by a camera from the data collection device 100 .

With the following configuration, the input/output unit 210 may receive a signal from a user through a user interface (UI) or output an operation result to the outside. Specifically, the input/output unit 210 may receive inputs of setting values for generating direction information, setting values for generating learning data, setting values for generating maps, and the like, or may output generated maps or learning data. there is.

With the following configuration, the storage unit 215 may store point cloud data or images received from the data collection device 100 and generated map or learning data. Specifically, the storage unit 215 may store an algorithm for generating direction information, an algorithm for generating learning data, an algorithm for generating a map, and the like.

With the following configuration, the data collection unit 220 may control to collect the point cloud data obtained by the LIDAR and the image taken by the camera from the data collection device 100 through the communication unit 205 . Specifically, the data collection unit 220 may control to receive point cloud data continuously acquired from LIDAR and to receive images photographed at preset intervals from a camera.

With the following configuration, the map generator 225 generates direction information based on point cloud data for traffic facilities detected on the image, so that the map can be effectively used with precision through the generated direction information.

Specifically, the map generator 225 may detect traffic facilities on the 2D image using a deep learning-based object detection algorithm. Here, the map generator 225 may generate direction information when the detected traffic facility is located within a preset distance or when the number of point groups included in the detected traffic facility is greater than or equal to a preset value. In addition, the map generator 225 may generate direction information when the ratio between the horizontal width and the vertical width of the detected bounding box of the traffic facility is equal to or less than a preset value. That is, in order to extract an accurate direction for the traffic facility, the map generator 225 determines the number of point clouds included in the detected traffic facility when the traffic facility is within a certain distance or the bounding box of the detected traffic facility is not imbalanced. Direction information can be extracted only when is greater than or equal to a certain level.

In this case, the map generator 225 may detect traffic facilities on the 2D image using distance-based Euclidean clustering. That is, when the map generator 225 extracts point cloud data for generating direction information, in order to prevent point cloud data of objects other than the detected traffic facility from being extracted, the map generator 225 determines the location of the traffic facility through the Euclidean clustering algorithm. You can configure a cluster for

In addition, the map generator 225 may perform down sampling by filtering the point cloud included in the detected traffic facility in the form of a voxel grid. That is, the map generator 225 may down-sample unbalanced point clouds among the point clouds included in the detected traffic facility in a balanced form.

In addition, the map generator 225 may accumulate point cloud data for detected traffic facilities and generate direction information when a predetermined specific value is satisfied. That is, the map generating unit 225 may generate direction information in a state in which sufficient point clouds for traffic facilities are secured by accumulating point clouds. For example, the map generator 225 may generate direction information after accumulating point cloud data for detected traffic facilities by a predetermined number of rotations of lidar.

In addition, the map generator 225 may obtain a representative plane by randomly searching point clouds for detected traffic facilities using a Random Sample Consensus (RANSAC) algorithm. That is, the map generator 225 may randomly select three points from the point group for the detected traffic facility and obtain a plane passing through the selected three points. In this case, inlier points having a distance from the plane smaller than a preset value may be identified, and the number of inlier points may be obtained. In this case, the map generator 225 may select a plane having the greatest number of inlier points as a representative plane.

In addition, the map generator 225 may calculate a normal vector extending vertically from the plane using a plane equation based on the representative plane. That is, the map generator 225 may obtain direction information on a representative plane through a plane equation described in the following equation.

[Equation 1]

Ax + By + Cz + D = 0

(Where (A, B, C) is the normal vector for the representative plane, D is the length of the normal vector, and (x, y, z) can be any point on the representative plane.)

Also, the map generator 225 may generate a map including direction information about the created traffic facility. In this case, the map generator 225 may display the generated normal vector on the map or include it in meta information of the map.

For example, a map containing direction information selectively provides information to a driver who needs information on traffic facilities pointing in a specific direction, or when the direction of a traffic facility is changed compared to the initially installed direction to a manager who manages traffic facilities We can assist you in determining that maintenance is required.

Meanwhile, existing modeling techniques perform modeling by scanning all point cloud data at once. However, since point cloud data collected from LIDAR installed in an airborne device has a lower density of point clouds toward the lower part of the object, conventional modeling techniques are not suitable for map generation.

Accordingly, the map generator 225 projects the top-view of the point cloud data obtained from the lidar installed in the airborne device, recognizes the edge of the object and scans it in a direction perpendicular to the ground, thereby increasing the accuracy of 3D modeling. can

Specifically, the map generator 225 may detect a reference area of the projected point cloud data, and may detect an edge of an object while scanning in a direction perpendicular to the ground based on the reference area. Here, the reference region may be the edge of the lowermost surface of the object included in the point cloud data. That is, the map generator 225 may detect the reference area, which is the lowermost surface of the object, and detect the edge of the object while scanning from the reference area in the upper direction of the object.

To this end, the map generator 225 may extract a closed curve from a point cloud located within a preset distance among the scanned point cloud data and detect the closed curve as an edge of the object. For example, the map generator 225 may detect an edge of an object based on an alpha shape algorithm. That is, the map generator 225 may form a border of an object surrounded by a plurality of circles by drawing a circle having a radius а connecting the contours of two randomly selected points from a point cloud located within a preset distance among the scanned point cloud data. there is.

In addition, the map generator 225 may detect borders detected before the edge of which the difference occurs from the reference area, when the width of the border detected based on the area of the reference area differs by more than a preset value.

In addition, the map generator 225 may model an area from the reference area to an edge detected before the edge where the difference occurs. At this time, the map generator 225 may update the border where the difference occurs from the previous reference area to a new reference area.

Here, the map generator 225 may model the object by connecting a point group included in a border detected before the edge where the difference occurs from the reference area into a triangle, but maximizing the minimum value of the interior angle of the triangle. there is. That is, the map generator 225 may model the object through delaunay triangulation. In other words, the map generator 225 constructs a super triangle including all point groups of the detected border, and if there are no other points in the circle containing the three points for each point in the initial triangle, We can add that triangle as a Delaunay triangle. At this time, the initial triangle can be deleted after all the Delaunay triangular network construction processes are performed. Then, the map generator 225 may form an external surface by meshing the modeled object.

In addition, for an object whose upper area is larger than its lower area, the map generator 225 may model the object by setting the uppermost edge of the object as the reference area and scanning the lower part of the object with the reference area as the center.

In addition, the map generator 225 projects the point cloud data onto the image, filters the projected point cloud data on the image based on distance, and generates a color map, so that it is expressed as being occluded by a specific object on the image. On the point cloud data, it is possible to effectively remove noise applied with the wrong color of the image with respect to the recognized area.

Specifically, the map generating unit 225 may generate unit point cloud data by grouping the point cloud data based on the movement distance of the lidar, and project the generated unit point cloud data onto an image captured within the movement distance. At this time, the map generator 225 may project the generated unit point cloud data onto an image captured at an intermediate point of the movement distance. For example, the map generator 225 may generate a virtual event reference point by calculating a distance based on GPS data obtained by a GPS receiver installed together with a lidar and a camera. In this case, when the virtual event reference points based on the distance are the 10m point and the 20m point, the map generator 225 may select an image acquired by the camera at the 15m point, which is a value between the 10m point and the 20m point. Here, the grouped unit point cloud data may include location information based on the world coordinate system.

In addition, the map generator 225 may project unit point cloud data onto an image table including information about a camera that has taken an image, a horizontal axis pixel (x), and a vertical axis pixel (y) of the image. That is, the map generator 225 may store information on points matching each pixel of the image on the image table. Here, the information on the point may include position information (x, y, z) based on lidar, an intensity value, matched color information (r, g, b), and position information based on the world coordinate system.

In addition, the map generator 225 may detect the distance of a point by using location information of a lidar criterion among information on a point per pixel of the image table. In this case, when information on a plurality of points is stored in one pixel, the map generator 225 may delete the remaining points except for a point having the closest distance based on lidar among the plurality of points. That is, when there are a plurality of points in one pixel, the map generator 225 determines that the remaining points except for the nearest point are points corresponding to the hidden area on the image and deletes them.

In addition, the map generation unit 225 compares distances between points existing in neighboring pixels by a preset number of pixels with respect to the filtered pixel, and compares the distance between points that exist in neighboring pixels to the pixel on which the filtering is performed. can be removed.

In addition, the map generator 225 may generate a color map based on intensity values of points included in the filtered image table, color information, and location information based on the world coordinate system.

In addition, the map generator 225 separates points having a preset color from point cloud data to which the color of the image is assigned, corrects the color of the separated points based on the colors of neighboring points, and generates a color map, Noise in which a bright color is applied to an object having a specific dark color due to the color of the sky or light reflection can be effectively removed.

Specifically, the map generator 225 may assign a color to an image located at the same coordinates as each point included in the point cloud data. That is, the map generator 225 may generate unit point cloud data by grouping the point cloud data based on the movement distance of the lidar, and project the generated unit point cloud data onto an image captured within the movement distance. In addition, the map generator 225 may generate a color map based on the intensities of the projected points, color information, and location information based on the world coordinate system.

Also, the map generator 225 may separate points having a preset color from among points having a specific height or higher with respect to the ground. At this time, the map generator 225 may generate a histogram based on the brightness value of the color-applied point cloud data, and detect points having an intensity value greater than or equal to a preset value. In addition, the map generator 225 may separate points having a preset color, saturation lower than a preset value, and brightness higher than a preset value from the color-applied point cloud data.

For example, the map generation unit 225 may correct the color of the sky on the map or the bright color of trees due to light reflection on the map with precision. To this end, the map generator 225 may separate points having a preset color from among points having a specific height or higher with respect to the ground so as not to affect the lane. At this time, the map generator 225 has a preset color from the color-applied point cloud data, but has a saturation lower than the preset value and a preset brightness so that only points corresponding to the color of the sky or the reflection of light can be separated. Points higher than the value can be isolated. For example, the map generator 225 may separate points having a Hue value of 160 to 300 that may correspond to the color of the sky. At this time, if the dots having a Hue value of 160 to 300 are separated, the dots included in the blue-based traffic facility may also be separated. Accordingly, the map generator 225 separates points having a Hue value of 160 to 300, but may exclude points having an intensity value higher than a preset value. That is, the map generator 225 may exclude points corresponding to traffic facilities based on the intensity value by using that a sign having a flat shape has a high intensity value. Here, the intensity value may be signal strength of LIDAR. For example, the map generator 225 may exclude points having an intensity value of 250 or more among points having a Hue value of 160 to 300.

Also, the map generator 225 may generate a color map after removing the separated points. However, the map generator 225 is not limited thereto, and the map generator 225 may correct the color of each separated point to the color of a point closest to the separated point except for the separated point. For example, the map generator 225 may correct the color of each separated point to an average color value of points located within a preset distance from the separated point except for the separated point. At this time, the map generator 225 may detect a point closest to the separated point using an octree algorithm.

With the following configuration, the learning data generating unit 230 may process the map generated by the map generating unit 225 into learning data. For example, the learning data generator 230 may process the generated map into learning data to be used for autonomous driving.

In addition, the learning data generation unit 230 may separately generate learning data for traffic facilities in order to increase the learning recognition rate for traffic facilities. In this case, the learning data generation unit 230 may generate the generated learning data for the map and the learning data for the traffic facility as one learning data, and provide it to the outside.

That is, the learning data generation unit 230 may generate learning data by processing a traffic facility sample image provided by the Korea Road Traffic Authority to be similar to an image actually captured by a camera.

To this end, the learning data generation unit 230 collects sample traffic facility images, and generates at least one processed image processed so that the collected sample traffic facility images can be recognized as images captured by a camera installed in a vehicle, Learning data may be generated by inserting a background into at least one generated processed image.

Specifically, the learning data generation unit 230 may generate a plurality of first processed images by applying shearing to the collected traffic facility sample images within a specific angular range. In this case, the learning data generation unit 230 may apply distortion to the collected traffic facility sample images with a random value between -10 and 10°. That is, the learning data generation unit 230 may apply distortion to the traffic facility sample image within a specific angle range in order to generate various types of learning data according to the direction and angle captured by the camera. In this case, the traffic facility sample image may be a front view of the traffic facility.

Also, the learning data generator 230 may generate a second processed image by applying Gaussian blur to the generated first processed image. In this case, the learning data generation unit 230 may randomly apply Gaussian blur to the first processed image within a preset radius value for a pixel. That is, the learning data generation unit 230 may apply Gaussian blur to generate various types of learning data that appear clear or blurry according to the shooting environment of the camera.

Also, the learning data generation unit 230 may generate a plurality of third processed images by applying an intensity value to the second processed image. Here, the intensity value may be brightness intensity. That is, the learning data generator 230 may randomly apply an intensity value within a preset intensity value range to the second processed image. That is, the learning data generating unit 230 may randomly apply an intensity value to generate learning data having various intensity values according to the shooting environment of the camera. In this case, the learning data generator 230 may selectively apply an intensity value to a specific region of the second processed image. That is, since a situation in which intensity values are different in various regions may exist even in one traffic facility image, intensity values may be selectively applied to specific regions.

Also, the learning data generation unit 230 may generate learning data by inserting a background into at least one generated processed image. Here, the background may be a random area on a plurality of 2D images collected through a camera mounted in a vehicle. In this case, the learning data generation unit 230 may generate learning data by randomly applying the size of the processed image to the background. In addition, the learning data generation unit 230 may generate learning data by randomly applying the position of the processed image to the background.

In this way, the learning data generation unit 230 processes the traffic facility sample image so that it can be recognized as an image actually captured by the camera and inserts a background into the processed image to generate learning data. Sufficient learning data can be secured for the facility.

As described above, hardware for implementing the logical components of the data generating apparatus 200 according to an embodiment of the present invention will be described in more detail.

3 is a hardware configuration diagram of a data generating device according to an embodiment of the present invention.

As shown in FIG. 3, the data generating device 200 includes a processor 250, a memory 255, a transceiver 260, an input/output device 265, and a data bus Bus, 270) and storage (Storage, 275) can be configured to include.

The processor 250 may implement operations and functions of the data generating device 200 based on instructions according to the software 280a in which the method according to the embodiments of the present invention is resident in the memory 255 . Software 280a in which a method according to embodiments of the present invention is implemented may be loaded in the memory 255 . The transceiver 260 may transmit and receive data to and from the data collection device 100 . The input/output device 265 may receive data required for the functional operation of the data generating device 200 and output data generation results. The data bus 270 is connected to the processor 250, the memory 255, the transceiver 260, the input/output device 265, and the storage 275, and is a movement path for transferring data between each component. role can be fulfilled.

The storage 275 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 280a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 275 may store software 280b in which a method according to embodiments of the present invention is implemented. Also, the storage 275 may store information necessary for performing a method according to embodiments of the present invention.

According to one embodiment of the present invention, the software (280a, 280b) for implementing a method of generating a map including direction information resident in the memory 255 or stored in the storage 275 is provided by the processor 250 fixed to the vehicle. Receiving at least one image captured by at least one installed camera, detecting a traffic facility on the received at least one image, using lidar for the detected traffic facility It may be a computer program recorded on a recording medium to execute the steps of generating direction information based on the acquired point cloud data and generating a map including the generated direction information.

According to an embodiment of the present invention, the software 280a, 280b for implementing a method of generating learning data for traffic facilities resident in the memory 255 or stored in the storage 275 is provided by the processor 250, pre-stored Collecting traffic facility sample images; generating at least one processed image so that the collected traffic facility sample images can be recognized as images taken from a camera installed in a vehicle; In order to execute the step of generating learning data by inserting a background into a processed image, it may be a computer program recorded on a recording medium.

According to one embodiment of the present invention, the software 280a, 280b for implementing a method of generating a map using an airborne lidar resident in the memory 255 or stored in the storage 275 is provided by the processor 250, the airborne device. Receiving point cloud data obtained from a lidar installed in , projecting a top-view of the received point cloud data, and scanning the projected point cloud data in a direction perpendicular to the ground ( In order to execute the step of generating a map by scanning, it may be a computer program recorded on a recording medium.

According to an embodiment of the present invention, the software 280a, 280b for implementing a projection-based color map generation method resident in the memory 255 or stored in the storage 275 is provided by the processor 250, the camera Receiving an image captured from ) and point cloud data obtained from a lidar, projecting the received point cloud data onto the image, and projecting the projected point cloud data onto the image based on a distance It may be a computer program recorded on a recording medium to execute the filtering step and the step of generating a color map by assigning the color of the image to the filtered point cloud data.

According to one embodiment of the present invention, the software 280a, 280b for implementing a method of generating a color map to which color correction is applied resides in the memory 255 or stored in the storage 275, the processor 250 is a lidar assigning a color of an image captured by a camera to point cloud data obtained from a lidar, separating a point having a preset color from the color-added point cloud data, and separating the separated point color It may be a computer program recorded on a recording medium in order to execute the step of generating a color map by correcting based on the color of neighboring points.

More specifically, the processor 250 may include an Application-Specific Integrated Circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired/wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a Liquid Crystal Display (LCD), an Organic LED (OLED), and/or a liquid crystal display (LCD). Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module may reside in memory 255 and be executed by processor 250 . The memory 255 may be internal or external to the processor 250 and may be connected to the processor 250 by various well-known means.

Each component shown in FIG. 3 may be implemented by various means, eg, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

Hereinafter, a method for generating a map including direction information according to an embodiment of the present invention will be described in detail.

4 is a flowchart illustrating a method of generating a map including direction information according to an embodiment of the present invention.

Referring to FIG. 4 , in step S110, the data generating device may receive an image captured by a camera.

Next, in step S120, the data generating device may detect traffic facilities on at least one received image. In this case, the data generating device may detect traffic facilities on the image using a deep learning-based object detection algorithm. Here, in order to extract an accurate direction for the traffic facility, the data generating device is provided when the traffic facility is within a certain distance, the bounding box of the detected traffic facility is not imbalanced, and the number of point clouds included in the detected traffic facility is greater than or equal to a certain distance. In this case, direction information may be extracted in step S130.

Next, in step S130, the data generating device may generate direction information based on point cloud data acquired by lidar corresponding to the detected traffic facility. In this case, the data generating apparatus may detect traffic facilities on the 2D image using distance-based Euclidean clustering. That is, when extracting point cloud data for generating direction information, the data generating device generates clusters for traffic facilities through a Euclidean clustering algorithm in order to prevent point cloud data of objects other than the detected traffic facility from being extracted. can be configured.

In addition, the data generating apparatus may perform down sampling by filtering the point cloud included in the detected traffic facility in the form of a voxel grid. That is, the data generating apparatus may down-sample unbalanced point clouds among the point clouds included in the detected traffic facility into a balanced form.

Also, the data generating device may accumulate point cloud data for the detected traffic facility and generate direction information when a predetermined specific value is satisfied. That is, the data generating device may generate direction information in a state in which sufficient point clouds for traffic facilities are secured by accumulating point clouds.

In addition, the data generating apparatus may obtain a representative plane by randomly searching the point cloud for the detected traffic facility using a random sample consensus (RANSAC) algorithm. That is, the data generating device randomly selects three points from the point cloud for the detected traffic facility and obtains a plane passing through the selected three points. At this time, inlier points having a distance from the plane smaller than a preset value may be identified, and the number of inliers may be obtained. In this case, the data generating device may select a plane having the largest number of inliers as a representative plane.

In addition, the data generating device may calculate a normal vector extending vertically from the plane using a plane equation based on the representative plane.

And, in step S140, the data generating device may generate a map including direction information about the generated traffic facilities. In this case, the data generating device may display the generated normal vector on the map or include it in meta information of the map.

Hereinafter, a method for generating learning data for traffic facilities according to an embodiment of the present invention will be described in detail.

5 is a flowchart illustrating a method of generating learning data for a traffic facility according to an embodiment of the present invention.

Referring to FIG. 5 , in step S210 , the data generating device may collect sample images of traffic facilities. Here, the traffic facility sample image may be a front view of traffic facilities provided by the Road Traffic Authority.

Next, in step S220, the data generating device may generate at least one processed image by processing the collected traffic facility sample image to be recognized as an image captured by a camera installed in the vehicle.

Specifically, the data generating device may generate a plurality of first processed images by applying shearing to the collected traffic facility sample images within a specific angular range. In this case, the data generating device may apply distortion to the collected traffic facility sample image with a random value between -10 and 10°. That is, the data generating device may apply distortion to the traffic facility sample image within a specific angular range in order to generate various types of learning data according to directions and angles photographed by the camera.

Also, the data generating apparatus may generate a second processed image by applying Gaussian blur to the generated first processed image. In this case, the data generating device may randomly apply Gaussian blur to the first processed image within a preset radius value for each pixel. That is, the data generating device may apply Gaussian blur to generate various types of learning data that appear clear or blurry according to the shooting environment of the camera.

Also, the data generating apparatus may generate a plurality of third processed images by applying an intensity value to the second processed image. That is, the data generating device may randomly apply an intensity value within a preset intensity value range to the second processed image. The data generating device may randomly apply an intensity value to generate learning data having various intensity values according to a photographing environment of a camera. In this case, the data generating device may selectively apply an intensity value to a specific region of the second processed image. That is, since a situation in which intensity values are different in various regions may exist even in one traffic facility image, intensity values may be selectively applied to specific regions.

And, in step S230, the data generating device may generate learning data by inserting a background into at least one generated processed image. Here, the background may be a random area on a plurality of 2D images collected through a camera mounted in a vehicle. In this case, the data generating device may generate learning data by randomly applying the size of the processed image to the background. In addition, the data generating device may generate training data by randomly applying a position of a processed image to a background.

Hereinafter, a method for generating a map using an aerial lidar according to an embodiment of the present invention will be described in detail.

6 is a flowchart illustrating a method of generating a map using an aerial lidar according to an embodiment of the present invention.

Referring to FIG. 6 , in step S310 , the data generating device may receive point cloud data on the ground acquired from a LIDAR installed in the airborne device.

Next, in step S320, the data generating apparatus may project a top-view of the received point cloud data. That is, the data generating device may generate point cloud data at a point of view when the ground is viewed vertically with respect to the airborne device.

In step S330, the data generating device may generate a map by scanning the projected point cloud data in a direction perpendicular to the ground.

Specifically, the data generating apparatus may detect a reference area of the projected point cloud data and detect an edge of the object while scanning in a direction perpendicular to the ground based on the reference area. Here, the reference region may be the lowermost surface of an object included in the point cloud data. That is, the data generating apparatus may detect the reference area, which is the lowermost surface of the object, and detect the edge of the object while scanning from the reference area toward the top of the object.

To this end, the data generating apparatus may extract a closed curve from a point cloud located within a preset distance among the scanned point cloud data and detect the closed curve as an edge of the object.

In addition, the data generating apparatus may detect borders detected before the edge of which the difference occurs from the reference area when the difference between the width of the detected edge and the width of the reference area is equal to or greater than a preset value.

Also, the data generating apparatus may model an area from the reference area to an edge detected prior to the edge where the difference occurs. At this time, the data generating device may update the border where the difference occurs from the previous reference area to a new reference area.

Here, the data generating apparatus may model the object by connecting a point cloud included in an edge detected prior to an edge that has a difference from the reference area in a triangle, so that the minimum value of the interior angle of the triangle is maximized. That is, the data generating device may be modeled through Delaunay triangulation. In other words, the data generating device constructs a super triangle including all point groups of the detected border, and if no other points exist within the circle containing the three points for each point in the initial triangle, the triangle is formed. It can be added as a Delaunay triangle. At this time, the initial triangle can be deleted after all the Delaunay triangular network construction processes are performed. Thereafter, the data generating device may mesh the modeled object to form an external surface.

Hereinafter, a method for generating a projection-based color map according to an embodiment of the present invention will be described in detail.

7 is a flowchart illustrating a projection-based color map generation method according to an embodiment of the present invention.

Referring to FIG. 7 , in step S410, the data generating apparatus may receive an image taken from a camera and point cloud data obtained from a lidar.

Next, in step S420, the data generating device may project the received point cloud data onto an image.

Specifically, the data generating apparatus may generate unit point cloud data by grouping the point cloud data based on the moving distance of the lidar, and project the generated unit point cloud data onto an image captured within the moving distance. In this case, the data generating device may project the generated unit point cloud data onto an image captured at an intermediate point of the moving distance. Here, the grouped unit point cloud data may include location information based on the world coordinate system. That is, the data generating apparatus may project the unit point cloud data onto an image table including information about a camera that has captured the image, a horizontal axis pixel (x), and a vertical axis pixel (y) of the image. That is, the data generating device may store information on points matching each pixel of the image on the image table. Here, the information on the point may include position information (x, y, z) based on lidar, an intensity value, matched color information (r, g, b), and position information based on the world coordinate system.

Next, in step S430, the data generating apparatus may filter the point cloud data projected onto the image based on the distance.

Specifically, the data generating apparatus may detect the distance of a dot by using positional information of a lidar reference among information on a dot per pixel of an image table. In this case, when information on a plurality of points is stored in one pixel, the data generating device may delete the remaining points except for the point having the closest distance based on lidar among the plurality of points.

Also, the data generating apparatus may compare distances between neighboring pixels by a predetermined number of pixels based on a point having the closest distance, and may remove a pixel having a distance greater than a predetermined value.

In step S440, the data generating device may generate a color map based on the intensity values of the points included in the filtered image table, color information, and location information based on the world coordinate system.

Hereinafter, a method for generating a color map to which color correction is applied according to an embodiment of the present invention will be described in detail.

8 is a flowchart illustrating a method for generating a color map to which color correction is applied according to an embodiment of the present invention.

Referring to FIG. 8 , in step S510, the data generating apparatus may assign a color of an image captured through a camera to point cloud data obtained from a lidar.

Specifically, the data generating apparatus may generate unit point cloud data by grouping the point cloud data based on the moving distance of the lidar, and project the generated unit point cloud data onto an image captured within the moving distance. In addition, the data generating apparatus may generate a color map based on intensities of projected points, color information, and location information based on a world coordinate system.

Next, in step S510, the data generating device may separate points having a preset color from among points having a specific height or higher with respect to the ground. In this case, the data generating apparatus may generate a histogram based on the brightness value of the color-applied point cloud data, and detect a point having an intensity value greater than or equal to a preset value. In addition, the data generating apparatus may separate points having a preset color, but having a saturation lower than a preset value and a brightness higher than the preset value from the color-assigned point cloud data.

And, in step S530, the data generating device may generate a color map after removing the separated points. However, the present invention is not limited thereto, and the data generating apparatus may correct the color of each separated dot to the color of a dot closest to the separated dot except for the separated dot.

9 and 10 are exemplary diagrams for explaining a method of generating a map including direction information according to an embodiment of the present invention.

As shown in FIG. 9 , the data generating device may detect traffic facilities from a road image captured by a camera. For example, the data generating device may recognize traffic facilities through YOLO, a detection algorithm using a deep learning method. That is, the data generating device may detect a bounding box including traffic facilities through the YOLO algorithm and additionally detect traffic facilities within the detected bounding box.

In addition, the data generating device may generate direction information about a traffic facility by using the detected traffic facility image and the point cloud data acquired by the LIDAR at the same time.

At this time, the data generating apparatus does not extract direction information when the number of point cloud data included in the detected traffic facility is insufficient or the bounding box is unbalanced as in (A), and the bounding box is not unbalanced as in (B). direction information of a traffic facility may be generated when the number of point cloud data is greater than or equal to a preset value.

Through this, as shown in FIG. 10, the data generating device generates direction information based on the point cloud data for traffic facilities detected on the image, so that the map can be effectively used with precision through the generated direction information. there is.

For example, maps with high precision, including direction information, can selectively provide information to drivers who need information about traffic facilities pointing in a specific direction, or to managers who manage traffic facilities when the direction of a traffic facility is determined compared to the originally installed direction. If it is changed, we can support you to determine that maintenance is required.

11 to 13 are exemplary diagrams for explaining a method of generating learning data for traffic facilities according to an embodiment of the present invention.

As shown in FIG. 11, although crawling has been performed for traffic facilities from a road image, there is a problem in that the amount of data is small and the traffic facility area must be directly cut out. In addition, although there are many traffic facilities such as 'speed limit', it was confirmed that it is difficult to obtain rare traffic facilities such as 'passing priority' and 'bicycles are allowed' even through crawling.

Accordingly, as shown in FIG. 12, a synthetic image to which only distortion was applied was created based on the sample image of traffic facilities provided by the Korea Road Traffic Authority to learn a classification model, but the classification accuracy was low due to a large difference from the image of the actual environment. confirmed that

Therefore, in order to increase the similarity between the traffic facility sample image and the image of the real environment, the map generating apparatus according to an embodiment of the present invention, as shown in FIG. 13, distorts and Gaussian blur in the traffic facility sample image A. And it is possible to generate a plurality of learning data as shown in (B) by generating a processed image by applying a random intensity value and inserting a background of a random area into the processed image.

In addition, in order to confirm the learning effect of the generated learning data, the same learning model (CoATNet) is obtained through learning data obtained by applying only distortion to the sample image of a traffic facility as shown in FIG. 12 and learning data generated according to an embodiment of the present invention. learned

As a result, the training data to which only warping was applied showed an accuracy of 8% in 246 test images, but the same test result after learning with training data to which the method of generating learning data for traffic facilities according to an embodiment of the present invention was applied It showed an accuracy of 66.2%.

Finally, a total of 42,920 pieces of learning data were generated for 107 signs by applying the learning data generation method for traffic facilities according to an embodiment of the present invention, and a classification model (EfficientNet V2 m) was trained.

As a result, when the learned classification model was evaluated for a total of 2,454 images (1,982 extracted from actual driving data, 472 obtained through crawling and cropping), it was confirmed that 96.169% accuracy was shown.

14 to 16 are exemplary diagrams for explaining a map generation method using an aerial lidar according to an embodiment of the present invention.

As shown in FIG. 14, the data generating device projects point cloud data A at the point of view of the ground vertically with respect to the airborne device, detects a reference area of the projected point cloud data, and based on the reference area While scanning in a direction perpendicular to the ground, the edge B of the object may be detected.

That is, as shown in FIG. 15 , the data generating apparatus may detect a reference area, which is the edge of the lowermost surface of the object, and detect the edge of the object while scanning from the reference area toward the top of the object.

In this case, when the width of the detected border with respect to the width of the reference area differs by more than a preset value, the data generating device may detect borders detected before the border where the difference occurs from the reference area.

Also, the data generating apparatus may model an area from the reference area to an edge detected prior to the edge where the difference occurs. At this time, the data generating device may update the border where the difference occurs from the previous reference area to a new reference area.

Here, as shown in (A) of FIG. 16 , the data generating apparatus may model the object through Delaunay triangulation. In other words, the data generating device constructs a super triangle including all point groups of the detected border, and if no other points exist within the circle containing the three points for each point in the initial triangle, the triangle is formed. It can be added as a Delaunay triangle.

Then, as shown in (B) of FIG. 16 , the data generating device may mesh the modeled object to form an external surface.

17 and 18 are exemplary diagrams for explaining a projection-based color map generation method according to an embodiment of the present invention.

As shown in (A) of FIG. 17 , the point cloud data detected by the laser pulse emitted in the direction a is generated in a part of the road covered by the parking prohibition sign. However, the point cloud data detected by the laser pulse emitted in the b direction includes point cloud data for an area covered in the a direction.

Accordingly, when point cloud data collected in directions a and b are projected onto an image captured by a camera in direction a, noise is generated in which the color of the parking prohibition sign is assigned to the area covered by the parking prohibition sign. do.

Accordingly, the present invention projects point cloud data onto an image, filters the projected point cloud data on the image based on a distance, and generates a color map, so that areas that are represented as occluded by a specific object on the image but recognized on the point cloud data For , as shown in (B), it is possible to effectively remove noise to which the wrong color of the image is applied.

In addition, as shown in (A) of FIG. 18 , it can be confirmed that noise is generated by a region hidden on the image behind the trash can. However, when filtering according to the embodiment of the present invention is applied, it can be confirmed that noise in the covered area is removed as shown in (B).

19 and 20 are flowcharts illustrating a method of generating a color map to which color correction is applied according to an embodiment of the present invention.

A color map generation method using color correction according to an embodiment of the present invention separates a point having a preset color from point cloud data to which an image color is assigned, and determines the color of the separated point based on the color of a neighboring point. By generating a color map by correcting it, it is possible to effectively remove noise in which a bright color is applied to an object having a specific dark color due to the color of the sky or light reflection.

That is, as shown in FIG. 19, it is possible to detect noise points due to sky color or light reflection as shown in (B) by separating points having a preset color from point cloud data (A) to which an image color is assigned. there is.

In addition, the map generating device may generate a color map as shown in FIG. 20 by detecting a point closest to the detected noise points using an octree algorithm and correcting the noise point with the color of the nearest point. there is.

Meanwhile, (A) of FIG. 21 is a diagram showing a color map before applying a color map generation method to which color correction is applied according to an embodiment of the present invention, and (B) is a diagram showing a color map after application.

As shown, when noise points due to the color of the sky or light reflection are separated, there is a problem in that the color of the points of the traffic facility having a blue system is also changed.

Accordingly, the map generating apparatus may exclude points having an intensity value higher than a preset value among the detected noise points from the noise points by using that the intensity value of the traffic facility is relatively high.

Referring to FIG. 22, as a result of generating a color map for a specific distance, in order to confirm the effect according to the present invention, noise points are effectively corrected after filtering (B) in comparison with the color map before filtering (A). I was able to confirm.

Referring to FIG. 23 , as a result of generating a color map for a specific distance in case of rain, it was confirmed that noise points were effectively corrected after filtering (B) compared to the color map before filtering (A).

In addition, referring to FIG. 24, as a result of checking the brightness distribution through the histogram for the first frame of the image for each distance in FIGS. I was able to confirm.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100: data collection device 200: data generating device
300: data generation system
205: communication unit 210: input/output unit
215: storage unit 220: data collection unit
225: map generation unit 230: learning data generation unit

Claims (10)

Receiving, by a map generator, at least one image captured by at least one camera fixedly installed in a vehicle;
detecting, by the map generating unit, a traffic facility on the received at least one image;
generating, by the map generating unit, direction information based on point cloud data obtained by a lidar corresponding to the detected traffic facility; and
generating, by the map generator, a map including the generated direction information; It is characterized by including,
The step of generating the direction information is
It is characterized in that the representative plane is obtained when the point cloud data for the detected traffic facility is accumulated and a specific value set in advance is satisfied,
The step of generating the direction information is
Characterized in that, the representative plane is obtained by accumulating the point cloud data for the detected traffic facility by a predetermined number of revolutions of the lidar, the map generation method including direction information.
The method of claim 1 , wherein detecting the traffic facility comprises:
A method for generating a map including direction information, characterized by detecting traffic facilities on the 2D image using a deep learning-based object detection algorithm. The method of claim 1, wherein generating the direction information
Characterized in that, generating direction information when the detected traffic facility is located within a preset distance, a map generating method including direction information. The method of claim 3, wherein generating the direction information
A method of generating a map including direction information, characterized in that generating direction information when the number of point groups included in the detected traffic facility is equal to or greater than a preset value. The method of claim 1 , wherein detecting the traffic facility comprises:
A method for generating a map including direction information, characterized by detecting traffic facilities on the 2D image using distance-based Euclidean clustering. The method of claim 5, wherein generating the direction information
A method for generating a map including direction information, characterized in that down sampling is performed by filtering the point cloud included in the detected traffic facility in the form of a voxel grid. 7. The method of claim 6, wherein generating the direction information
Characterized in that, a representative plane is obtained by randomly searching the point cloud for the detected traffic facility using a random sample consensus (RANSAC) algorithm. memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
receiving, by the processor, at least one image taken by at least one camera fixedly installed in the vehicle;
detecting, by the processor, a traffic facility on the received at least one image;
generating, by the processor, direction information based on point group data obtained by a lidar corresponding to the detected traffic facility; and
Including, executing, by the processor, generating a map including the generated direction information,
The step of generating the direction information is
It is characterized in that the representative plane is obtained when the point cloud data for the detected traffic facility is accumulated and a specific value set in advance is satisfied,
The step of generating the direction information is
A computer program recorded on a recording medium, characterized in that the representative plane is obtained by accumulating the point cloud data for the detected traffic facility by a predetermined number of revolutions of the lidar.
The method of claim 8, wherein generating the direction information
A computer program recorded on a recording medium, characterized in that for generating direction information when the detected traffic facility is located within a preset distance.
9. The method of claim 8, wherein detecting the traffic facility comprises:
A computer program recorded on a recording medium, characterized in that for detecting traffic facilities on the 2D image using distance-based Euclidean clustering.

Images