KR20220071112A - Method and apparatus for generating a map for autonomous driving and recognizing location - Google Patents

Abstract

Provided are a method for generating and managing a map for autonomous driving, and a method and a device for recognizing a position using the same. When generating a map, three-dimensional coordinates information corresponding to a three-dimensional space is projected on a two-dimensional plane to obtain a spherical distance image, and semantic division is performed with respect to the spherical distance image to generate a semantic division image. Moreover, map data including the spherical distance image, the semantic division image, and property information of a vehicle lane is generated.

Description

A map generation and management method for autonomous driving, and a method and apparatus for location recognition using the same

The present disclosure relates to autonomous driving, and more particularly, to a method and apparatus for generating a map for autonomous driving and recognizing a location based on the generated map.

In order for the autonomous vehicle to operate, the vehicle needs to know the precise location of the vehicle and the surrounding environment at each moment in order to plan a route and determine the driving method (maneuver) at every moment. In addition, for this, a high-precision map and an accurate location recognition technology based on the map are absolutely necessary.

Existing related methods include the following. First, there is a method of using a high definition (HD) map generated as a set of lines with lane-level precision. It recognizes the location through lane recognition and map comparison, and when creating a map, it creates map data at the lane level manually using a point cloud map. Accordingly, there is a problem in that there is a relatively high possibility that modeling of the environment is not accurate. Second, there is a method of using the 3D coordinate set itself called a point cloud as a map. In this method, an Iterative Closest Point (ICP) method and a Normal Distribution Transform (NDT) method are mainly used. Although this method can obtain the highest precision, it is sensitive to the initial value, and there is a high possibility that the location recognition will fail in a situation where there are no rich 3D features such as a highway. Moreover, a very large storage space is required to store the 3D point cloud, and there is a problem that a lot of calculations are required to search and process each point. Third, there is a method based on the intensity of the road surface pattern. In this method, by limiting the movement of the vehicle and the shape of the road in two dimensions, the accuracy is deteriorated on a hill road, etc., and the stability may be deteriorated in the section where the same floor pattern is repeated. Finally, a method based on a pattern around the road perpendicular to the road surface, also called RoadDNA. This method can save storage space compared to the 3D point cloud, but the precision may be lowered because there is no height information and road surface information.

An object of the present disclosure is to provide a method and apparatus for generating map data by shaping a pattern of a road surface, a pattern around a road, and three-dimensional shape information with one image.

Another object of the present disclosure is to provide a method and apparatus for generating a three-dimensional high-precision map using less data storage space.

In addition, an object of the present disclosure is to provide a method and an apparatus for more precisely recognizing a location using a corresponding map.

According to an embodiment of the present disclosure, a method for generating a map for autonomous driving is provided. The method may include: obtaining 3D coordinate information corresponding to a 3D space; obtaining a spherical distance image by projecting the three-dimensional coordinate information onto a two-dimensional plane; generating a semantic segmentation image by performing semantic segmentation on the spherical distance image; and generating map data including the spherical distance image, the semantic segmentation image, and attribute information of the lane.

In an embodiment, generating the semantic segmentation image may include: performing semantic segmentation on the spherical distance image; and removing a moving object from the image obtained by the semantic segmentation to generate the semantic segmentation image including road information. In this case, the road information is used as segmentation information including a pattern of a road surface, a road pattern, and a pattern around the road, and the attribute information of the road may be obtained using the segmentation information included in the semantic segmentation image. have.

In one embodiment, the map has a graph-based structure, the map data is mapped for each node constituting the graph, the edge information between the nodes includes posture information, and the posture information is the spherical distance image of the spherical distance image. It may be a result of motion and posture estimation using the gradient or the gradient of the semantic segmented image.

In one embodiment, the generating of the map data may include: acquiring map data of a neighboring node of the node in a state in which map data of a node corresponding to a map data generating point is generated; and synthesizing the map data of the node and the map data of the neighboring nodes, and using them as map data for the node.

In an embodiment, the step of synthesizing the map data and using the map data as the map data for the node may include: synthesizing a spherical distance image of the node and a spherical distance image of the neighboring node; synthesizing the semantic segmented image of the node and the semantic segmented image of the neighboring node; generating vehicle attribute information based on the synthesized semantic segmentation image; and using data including the synthesized spherical distance image, the synthesized semantic segmentation image, and the lane attribute information as map data for the node.

In an embodiment, the synthesizing of the spherical distance image may include obtaining a synthesized spherical distance image based on a method of selecting an intermediate value from values accumulated for each pixel. In this case, the synthesizing of the semantic segmented image may acquire a synthesized semantic segmented image based on a method of selecting a value having the largest distribution among values accumulated for each pixel.

In one embodiment, when the spherical distance image is synthesized and the semantic segmented image is synthesized, the image of the node and the image of the neighboring node are coordinate-converted based on the node, and corresponding to the image of the node for each pixel 2D image data is obtained by accumulating and storing the pixel value and the corresponding pixel value of the image of the neighboring node, and selecting one of the accumulated values for each pixel, and for each pixel of the 2D image data , a method of selecting a final value of a corresponding pixel based on a result of comparing a value of a neighboring pixel with a value of the corresponding pixel, the synthesis may be performed.

In an embodiment, the method may further include compressing the map data. In this case, in the compressing step, the spherical distance image and the semantic segmented image are combined into one image for each pixel and stored. Distance information may be located, and segmentation information of the semantic segmented image may be located in a second bit of the set bits.

In an embodiment, the compressing may include: compressing the spherical distance image and the semantic segmented image; quantizing the distance information of the compressed spherical distance image, and locating the quantized distance information in a higher bit among set bits of a pixel; and quantizing the segmentation information of the compressed semantic segmentation image, and locating the quantized segmentation information in a lower bit among set bits of a pixel.

In an embodiment, the method may further include registering the map data. In this case, the registering may include: registering the generated map data as new map data when there is no existing map data in the surrounding coordinates of the generated map data; and updating the existing map data based on the generated map data when there is existing map data in the peripheral coordinates of the generated map data.

In an embodiment, the updating may include repeatedly performing an operation of storing the generated map data together with time information as additional map data; and comparing the stored additional map data and updating the existing map data based on the additional map data when different changes are maintained for a set time.

In an embodiment, the method may further include optimizing a graph constituting the map. In the optimizing step, when GPS data is obtained for each node, a GPS node having the GPS data is connected to the corresponding node, and when loop closure is detected in the graph, corresponding nodes are connected to each other, and each node using the posture information The attribute information of the lane included in the node can be combined.

According to another embodiment of the present disclosure, a method for recognizing a location based on a map for autonomous driving is provided. The method includes: obtaining a spherical distance image based on data input from a sensor at a current node, and generating a semantic segmentation image from the spherical distance image; acquiring an initial location based on GPS data or a previous driving route; retrieving map data from the map based on the initial location; and obtaining a current location based on the searched map data, wherein the map data of the map includes a spherical distance image, a semantic segmentation image, and attribute information of a lane.

In one embodiment, the map has a graph-based structure, the map data is mapped for each node constituting the graph, the edge information between the nodes includes posture information, and the posture information includes the spherical distance image It may be a result of motion and posture estimation using the slope of , or the slope of the semantic segmented image.

In one embodiment, the obtaining of the current location includes estimating the posture information of the current node with respect to the searched map data, and the estimated posture information and the posture information according to the movement of the current node satisfy a setting condition. In this case, the current location may be obtained using the map data, and if the current node has GPS data, the current location may be obtained by additionally using the GPS data.

According to another embodiment of the present disclosure, an apparatus for recognizing a location is provided. The device may include: an interface device; and a processor connected to the interface device configured to perform location recognition, wherein the processor projects 3D coordinate information corresponding to a 3D space input through the interface device onto a 2D plane to obtain a spherical distance image action to do; generating a semantic segmented image by performing semantic segmentation on the spherical distance image; acquiring an initial location based on GPS data or a previous driving route; retrieving map data from a map based on the initial location; and acquiring a current location based on the searched map data, wherein the map data of the map includes a spherical distance image, a semantic segmentation image, and attribute information of a lane.

In one embodiment, the map has a graph-based structure, the map data is mapped for each node constituting the graph, the edge information between the nodes includes posture information, and the posture information includes the spherical distance image It may be a result of motion and posture estimation using the slope of , or the slope of the semantic segmented image.

In an embodiment, the processor is further configured to generate map data including the spherical distance image, the semantic segmentation image, and the attribute information of the lane, and the processor is configured to: acquiring map data of nodes adjacent to the node in a state in which map data of a node corresponding to a map data generating point is generated when an operation is performed; synthesizing the spherical distance image of the node and the spherical distance image of the neighboring node; synthesizing the semantic segmented image of the node and the semantic segmented image of the neighboring node; generating vehicle attribute information based on the synthesized semantic segmentation image; and using data including the synthesized spherical distance image, the synthesized semantic segmentation image, and the attribute information of the lane as map data for the node.

In an embodiment, when the processor performs the operation of generating the semantic segmentation image, performing semantic segmentation on the spherical distance image; and removing a moving object from the image obtained by the semantic segmentation to generate the semantic segmentation image including road information. In this case, the road information is used as segmentation information including a pattern of a road surface, a road pattern, and a pattern around the road, and the attribute information of the road may be obtained using the segmentation information included in the semantic segmentation image. have.

In one embodiment, the map data is compressed and stored and managed, and when compressed, the spherical distance image and the semantic segmented image are combined into one image for each pixel and stored, and for a pixel composed of setting bits, the setting bit The distance information of the spherical distance image may be located in a first bit of the spherical distance image, and segmentation information of the semantic segmented image may be located in a second bit of the setting bits.

According to embodiments, since map data can be generated by shaping a pattern of a road surface, a pattern around a road, and 3D shape information with a single image, a 3D high-precision map for autonomous driving, etc., can be generated using a small data storage space. can provide In particular, it is possible to generate and provide map data including an image in which distance information and semantic division information are projected on a spherical plane, corresponding to 3D coordinate information corresponding to 3D space, and attribute information of a road.

In addition, shape information (spherical distance image), pattern information (semantic segmentation image), and lane attribute information are generated as one map data, and based on this, the location of a vehicle, etc. can be easily recognized and automatically generated and update.

In addition, since map data is stored for each graph-based node, management of creation and update is advantageous.

1 is an exemplary diagram illustrating a three-dimensional map according to an embodiment of the present disclosure.
2A and 2B are exemplary diagrams illustrating map data of a node according to an embodiment of the present disclosure.
3 is a conceptual diagram illustrating an overall process of generating a map for autonomous driving and recognizing a location based on the map according to an embodiment of the present disclosure.
4 is a flowchart of a method of acquiring a spherical distance image according to an embodiment of the present disclosure.
5 is an exemplary diagram illustrating a spherical coordinate system according to an embodiment of the present disclosure.
6 is an exemplary view illustrating a spherical distance image according to an embodiment of the present disclosure.
7 is a flowchart of a method for obtaining a semantic segmented image according to an embodiment of the present disclosure.
8 is an exemplary diagram illustrating a result of performing semantic division according to an embodiment of the present disclosure.
9 is a flowchart of a motion and posture estimation method according to an embodiment of the present disclosure.
10 is a flowchart of a location recognition method according to an embodiment of the present disclosure.
11 is an exemplary diagram illustrating a posture graph according to an embodiment of the present disclosure.
12 is a flowchart of a method for generating map data according to an embodiment of the present disclosure.
13 is an exemplary diagram illustrating map data generation according to an embodiment of the present disclosure.
14 is an exemplary diagram illustrating compressed map data according to an embodiment of the present disclosure.
15 is an exemplary diagram illustrating optimization of map data according to an embodiment of the present disclosure.
16 is a diagram illustrating a structure of a location recognition apparatus according to an embodiment of the present disclosure.
17 is a diagram illustrating a structure of an apparatus for managing map data according to an embodiment of the present disclosure.
18 is a structural diagram illustrating a computing device for implementing a method according to an embodiment of the present disclosure.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

In this specification, expressions described in the singular may be construed in the singular or plural unless an explicit expression such as “a” or “single” is used.

In addition, terms including an ordinal number such as first, second, etc. used in an embodiment of the present disclosure may be used to describe the components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, a method and apparatus for generating a map for autonomous driving and recognizing a location based on the map according to an embodiment of the present disclosure will be described with reference to the drawings.

1 is an exemplary diagram illustrating a three-dimensional map according to an embodiment of the present disclosure.

In an embodiment of the present disclosure, map data for a 3D map obtained for autonomous driving has a node (N)-based graph structure as illustrated in FIG. 1 . Here, the node-based graph structure is expressed on a point cloud map, for example. Nodes of the graph are located at set intervals, and in particular, as shown in FIG. 1 , they may be generated at set intervals around an intersection. The edge information between each node is pose information between the nodes.

The three-dimensional map illustrated in FIG. 1 is an existing point cloud map, and consists of many three-dimensional points in order to express a three-dimensional shape. In some cases, characteristic information such as intensity of a point may be included in information about each point. These point cloud-based 3D maps enable precise location recognition by depicting the real environment in detail, but the data size is very large, so it is difficult to store, manage, and process it.

In an embodiment of the present disclosure, the map data of each node includes an image in which distance information and semantic division information are projected on a spherical plane, and attribute information of a road.

2A and 2B are exemplary diagrams illustrating map data of a node according to an embodiment of the present disclosure.

If the image acquired around one node and the vehicle speed property information are shaped, it is as exemplified in FIG. 2A. The map data of such a node includes a spherical range image, a semantic segmentation image, and attribute information of a lane.

The spherical distance image is an image obtained by projecting three-dimensional coordinate information onto a spherical plane instead of a three-dimensional point around a node, as illustrated in (a) of FIG. 2B . This spherical distance image includes distance information. The distance information may be represented, for example, in different colors.

A semantic segmentation image is an image obtained by semantically segmenting a spherical distance image. The object to be divided in the image is a moving object and road information, the moving object includes a vehicle and a person, and the road information includes a road, a lane, a road surface, and the like. If a moving object is removed from the spherical distance image after semantic segmentation, as illustrated in FIG. A segmented image is acquired. Here, including the pattern of the road surface, the road pattern, and the pattern around the road, it may be collectively referred to as semantic segmentation information.

In the semantic segmentation image according to an embodiment of the present disclosure, texture information and semantic information may be expressed together with less data, unlike intensity information of a point.

The lane attribute information is information indicating the attribute of the lane generated from the semantic division information. The attribute information of the lane is obtained using road information obtained from semantic segmentation information and various additional information related to the lane (eg, lane limit speed, road mark, etc.), as illustrated in (c) of FIG. 2B As described above, the speed limit may be included for each lane.

The attribute information of the lane may include information obtained from color information of the lane. For example, in an image obtained from a camera, color information of each pixel corresponding to a pixel dividing a lane is calculated to obtain color information of a lane, and additional lane properties necessary for map generation can be obtained from the color information of the lane. .

In an embodiment of the present disclosure, map data of a node includes such a spherical distance image, a semantic segmentation image, and attribute information of a lane. In particular, the size of stored data is reduced by expressing 3D information as a 2D image, and not only a very fast operation is possible, but also restoration into a 3D shape is easy. The map data is mutually independent for each node, so that it is easy to search, modify, and manage the data.

In an embodiment of the present disclosure, the above map data is generated, a location is recognized based on the map data, and data management including storage and optimization of the map data is additionally performed.

3 is a conceptual diagram illustrating an overall process of generating a map for autonomous driving and recognizing a location based on the map according to an embodiment of the present disclosure.

The process according to the embodiment of the present disclosure may be largely divided into a location recognition part and a map management part, as shown in FIG. 3 , and may be performed. The location recognition part uses sensor data to recognize location or create new map data. Map data processing and optimization are performed in the map management part.

Specifically, in the location recognition part, for example, location recognition is performed or new map data generation is performed in real time while the vehicle is driving. First, a spherical range image SRI is generated by projecting 3D information collected from a sensor such as LiDAR or a camera onto a 2D plane (S10), and semantic segmentation is performed on the spherical distance image to A divided image is acquired (S11). Then, the motion of the vehicle is estimated using the spherical distance image and the semantic segmentation image (S12). Thereafter, location recognition for acquiring the location of the vehicle is performed based on the moving path of the vehicle, map data that has already been created, and global positioning system (GPS) information (S13).

When recognizing a location, if there is no map data or a large error at the current location, new map data is generated (S14). New map data may also be generated each time the vehicle passes a set area, eg, an intersection, or at set intervals while the vehicle is driving. As described above, the map data includes a spherical distance image, a semantic segmentation image, and lane attribute information.

In the map management part, map data newly generated in the location recognition part is registered (S20). When registering new map data, a new registration process is performed by comparing the new map data with the existing map data, or an update process of the existing map data is performed (S21).

In order to improve the precision of the stored map data, it is possible to detect a loop closure in the map data and perform posture graph optimization (S22 to S23) to determine the precise location of each node.

And, the attribute information of the lane included in each node may be updated using the optimized node location information (S24). That is, the high-resolution map data at the level of the road is updated by combining the optimized roadway location and location.

The process of this map management part can be performed in non-real time. In addition, in order to provide map data to several vehicles and collect map data, it may be designed to be performed on a separate server.

Here, a description will be given of recognizing the location of the vehicle as an example, but it is not necessarily limited to recognizing the location of the vehicle.

Next, a specific method of each process as described above will be described.

First, a method of acquiring a spherical distance image will be described.

4 is a flowchart of a method of acquiring a spherical distance image according to an embodiment of the present disclosure.

In an embodiment of the present disclosure, a spherical distance image is obtained by projecting 3D coordinate information corresponding to 3D spatial information onto a 2D plane. In particular, in an embodiment of the present disclosure, 3D coordinate information is projected onto a 2D surface using a spherical coordinate system so that data projection in all directions around the vehicle is possible. The 3D coordinate information may be collected from a sensor capable of acquiring 3D spatial information, such as a LiDAR or a stereo camera, and is also called a 3D point cloud.

To this end, three-dimensional coordinate information is obtained (S100), and the three-dimensional coordinate information is converted into a spherical coordinate system (S110).

5 is an exemplary diagram illustrating a spherical coordinate system according to an embodiment of the present disclosure.

가 구좌표계 상의 한 점

으로 변환될 수 있으며(도 5의 (a) 참조), 이를 수식으로 나타내면 다음과 같다. Based on the spherical coordinate system of FIG. 5, a point on the Cartesian coordinates

is a point in the spherical coordinate system

can be converted to (see Fig. 5 (a)), and expressed as a formula as follows.

은 거리(range or radial distance)를 나타내고,

는 방위각(azimuth)을 나타내며, 그리고

는 고도(altitude)를 나타낸다. 구 좌표계 상의 점을 2차원 표면에 투영한다(S120). 즉, 구 좌표계상의 한 점 m을 다음 수식을 사용하여 2차원 좌표계의 한 좌표(영상 좌표라고도 명명됨)로 변환한다. here,

represents the range or radial distance,

represents the azimuth, and

is the altitude. A point on the spherical coordinate system is projected on a two-dimensional surface (S120). That is, a point m on the spherical coordinate system is converted into one coordinate (also called image coordinate) of a two-dimensional coordinate system using the following equation.

값을 가지는 영상 좌표

로 변환된다(도 5의 (b) 참조). Through Equation 2 above, a point m on the spherical coordinate system is

image coordinates with values

is converted to (see Fig. 5 (b)).

,

는 각도와 화소 사이의 변환 비례 상수를 나타내며, 연산의 편의를 위해

로 정의될 수 있다. 본 개시의 실시 예에서는, 세로축의 좌표

를 구하기 위해

가 아닌

를 변수로 사용하는 것에 유의한다. 이에 따라, 위의 좌표 변환을 위한 수식(수학식 2)을 간략화할 수 있으며, 또한

가 큰 값의 해상도가 낮아지는 특성을 가진다. 이는 가까이 있는 바닥면이나 높은 건물 등 상대적으로 비중이 낮은 부분의 데이터를 축소하고 영상 데이터의 크기를 줄이는 역할을 한다. here

,

represents the conversion proportional constant between the angle and the pixel, for convenience of operation

can be defined as In an embodiment of the present disclosure, the coordinates of the vertical axis

to save

not

Note the use of as a variable. Accordingly, the formula (Equation 2) for the above coordinate transformation can be simplified, and also

It has a characteristic that the resolution of a large value decreases. This reduces the data of relatively low proportions, such as a nearby floor or a tall building, and reduces the size of the image data.

As described above, each point of the three-dimensional point cloud (three-dimensional coordinate information) is projected onto a two-dimensional surface through Equations 1 and 2 to generate a spherical distance image (S130).

Meanwhile, the restoration from the spherical distance image to the 3D coordinate information can be simply performed from the following inverse transformation equation.

및

는 수학식 2로부터,

,

로 정의되며, 이때, 화소 단위로 미리 계산된 테이블을 사용할 수 있다. Here, the variable

and

is from Equation 2,

,

, and in this case, a table calculated in advance in units of pixels may be used.

6 is an exemplary view illustrating a spherical distance image according to an embodiment of the present disclosure.

6A is an image in which a spherical distance image is expressed according to a distance, where different colors may be displayed according to a distance. 6 (b) and (c) show textures corresponding to spherical distance images. Specifically, (b) of FIG. 6 shows the intensity according to the signal reflected from the lidar sensor corresponding to the spherical distance image. 6C shows a camera image (eg, brightness) corresponding to a spherical distance image.

In this way, texture information such as intensity and brightness may be acquired together with distance information from the spherical distance image. Texture information has a weakness in that the intensity changes according to the influence of weather, illuminance, and the like and requires more storage space.

In an embodiment of the present disclosure, semantic segmentation information is used instead of texture information obtained from a spherical distance image.

Next, a method for acquiring a semantic segmented image will be described.

7 is a flowchart of a method for obtaining a semantic segmented image according to an embodiment of the present disclosure.

In an embodiment of the present disclosure, semantic division is dividing each pixel in an image into units of information that it means. The brightness information of an image is divided into several representative semantic units. Accordingly, it is robust to environmental changes and can be stored with very little data.

As shown in the attached FIG. 7 , a spherical distance image and a texture image corresponding thereto (eg, the intensity image of FIG. 6B ) are used as input ( S200 ). Semantic segmentation is performed using these input images (S210). When performing semantic segmentation, a deep neural network may be used.

Each pixel in the image is classified into units of information that it means, for example, moving object information and road information, to obtain a semantic segmented image (S220). Here, the moving object information indicates a moving object such as a vehicle or a pedestrian, and the road information indicates a road, a lane, a road surface display, and the like. Moving object information is used for obstacle processing during autonomous driving, and other road information is used as texture information.

Then, the final semantic segmentation image is obtained by removing the moving object from the image obtained in step S220 (S230).

8 is an exemplary diagram illustrating a result of performing semantic division according to an embodiment of the present disclosure.

8A shows a spherical distance image, and may be a color image displayed in different colors according to distance information. 8B shows an intensity image, ie, a texture image, according to a signal reflected from the lidar sensor corresponding to the spherical distance image. FIG. 8(c) shows an image as a result of performing semantic segmentation using a deep neural network with a spherical distance image and a texture image as inputs. FIG. 8(d) shows an image in which a moving object (vehicle, pedestrian, etc.) is removed from the result image of FIG. 8(c). Here, only road information remains, and this may be used as information on the pattern of the road surface, the road pattern, and the pattern around the road.

Location recognition of vehicles, etc. is performed based on such map data.

First, a method of performing motion and posture estimation for position recognition will be described.

9 is a flowchart of a motion and posture estimation method according to an embodiment of the present disclosure.

In an embodiment of the present disclosure, location recognition is performed using a spherical distance image/semantic segmentation image.

An important factor for generating a map or recognizing a location is motion estimation for estimating a moving path of a vehicle and estimating a vehicle's pose from landmarks on the map. Estimating the amount of geometric change that occurs between sensor data collected at continuous time is called motion estimation. Estimating the amount of change collected from different sensors or at different time points is called posture estimation. For motion estimation and posture estimation, it is necessary to solve the same problem of estimating parameters for the converted distance and angle between sensors.

An embodiment of the present disclosure provides a method of estimating self-motion estimation and posture on map data using a single method using a spherical distance image.

Specifically, ego-motion estimation and posture estimation are performed using a spherical distance image (or a semantic segmented image). For self-motion estimation, a method of estimating a motion obtained from an image change between a previous frame and a current frame, and obtaining an optimal parameter for this purpose may be used.

는 3차원 공간 상의 이동(

) 및 회전(

)이며, 총 6개의 파라미터를 갖는 벡터이다. 설명의 편의상, 추정해야 할 파라미터는 변위에 대한 파라미터라고 명명할 수 있다. For self-motion estimation and posture estimation on map data, parameters to be estimated in an embodiment of the present disclosure

is the movement in three-dimensional space (

) and rotation (

) and is a vector with a total of 6 parameters. For convenience of description, a parameter to be estimated may be referred to as a parameter for displacement.

가 제2 영상(예: 현재 프레임)의

에 대응(correspondence)된다고 하면, 이는 변환 함수

를 사용하여

로 표현할 수 있고, 두 대응 사이의 잔차(residual)는 다음 수식으로 정의될 수 있다.For example, data of the first image (eg, previous frame)

of the second image (eg, the current frame)

Assuming that it corresponds to , it is a transform function

use with

It can be expressed as , and the residual between the two correspondences can be defined by the following equation.

Finding a parameter value that minimizes the sum of the squares of the residuals is called least squares minimization, and it is expressed as an equation as follows.

를 사용하여 잔차를 처리한다.Various methods can be used to solve this equation. In the exemplary embodiment of the present disclosure, an M-estimation method having fast convergence and robust noise is used, but the present disclosure is not limited thereto. In the M-estimation method, in order to effectively process the noise, the loss function is

to handle the residuals.

을 갱신하는 것을 반복하는 것에 의해 이루어진다.Estimation of the parameter is the amount of change of the parameter until the error converges using the following equation

This is done by repeating updating

는 변환 함수

의 자코비안(Jacobian) 행렬이며,

는 손실 함수의 미분과 잔차의 곱인

로 정의되는 가중치(weight) 행렬이다. 연산자

는 유클리드 공간(Euclidean space)에 대한 정규 합(normal addition) 연산을 나타내며, 지수 사상(exponential map)을 사용하여 계산된다. 손실 함수로는 Huber, Tukey, t-distribution 등이 있으며, 추정 오차가 커지는 영역의 값을 감쇄하는 기능을 한다. here,

is the transform function

is the Jacobian matrix of

is the product of the derivative and residual of the loss function

It is a weight matrix defined by . Operator

represents a normal addition operation on a Euclidean space, and is calculated using an exponential map. The loss functions include Huber, Tukey, and t-distribution, and they function to attenuate the value in the region where the estimation error increases.

As described above, according to an embodiment of the present disclosure, a necessary parameter estimation is solved by obtaining a transform function and a Jacobian of a spherical distance image.

라고 하며, 변환 함수의 자코비안 행렬

는 체인 룰(chain rule)로 인해 다음과 같이 분해된다.The value of each pixel of the spherical distance image

, and the Jacobian matrix of the transform function

is decomposed as follows due to the chain rule.

은 구면 거리 영상의 기울기(gradient)를 나타낸다. 두 번째 항은 수학식 (2)의 편미분이며, 다음과 같이 구해진다.In Equation 8 above, the first term

denotes the gradient of the spherical distance image. The second term is the partial derivative of Equation (2), and is obtained as follows.

그리고

이다. 수학식 8에서, 마지막 항은 파라미터

를 탄젠트 공간(tangent space)의 Lie algebra로 정의하였을 때 다음과 같이 정의된다.here,

and

to be. In Equation 8, the last term is a parameter

When is defined as Lie algebra in tangent space, it is defined as follows.

Through the above process, it is possible to obtain the final Jacobian matrix in a relatively simple form as follows.

로 거리 정보뿐만 아니라, 의미론적 분할 영상을 사용할 수 있다. 수학식 11에서의 첫 번째 항

을 구면 거리 영상의 기울기 대신에, 의미론적 분할 영상의 기울기로 대체할 수 있다. When the Jacobian matrix is obtained in this way, a parameter for displacement can be calculated using Equations (4), (7), and (11) above. On the other hand, in this embodiment of the present disclosure, the value of each pixel

As such, not only distance information, but also semantic segmentation images can be used. The first term in Equation 11

can be replaced with the slope of the semantic segmented image instead of the slope of the spherical distance image.

When a semantic segmented image is used, separate processing may be performed. The semantic segmented image has a representative value for each pixel, and it is necessary to substitute an appropriate value for image processing. It is helpful to improve the precision by analyzing the precision of the division result and replacing it with a value proportional to it. In order to improve operation speed and robustness, it is preferable to select only important pixels rather than calculating the values of all pixels. First, pixels for a moving object are removed from the semantic segmentation result. Next, a value with an image slope greater than or equal to a certain threshold is selected, but pixels having a high spatial frequency, such as the leaves of a street tree, are excluded. In an embodiment of the present disclosure, there is no need for a technique such as an octree for processing three-dimensional coordinates, and data processing is possible at very high speed, so parameters of 0.6% or less in operation time of 5 ms or less in the latest CPU It is a very efficient method that can expect estimation error.

As described above, according to an embodiment of the present disclosure, referring to FIG. 9 , a slope of a spherical distance image or a semantic segmented image is obtained ( S300 ). And a Jacobian matrix is obtained using this gradient (S310), and the first image based on the equations (Equations (4), (7) and (11)) described above using the Jacobian matrix Estimate a parameter that minimizes the residual between (eg, previous spherical distance image) and the second image (eg, current spherical distance image) to estimate movement and posture according to movement and rotation in 3D space (S320) . The motion and posture estimation results obtained through this estimation process are called posture information for convenience of explanation. Thereafter, position recognition is performed using the posture information.

10 is a flowchart of a location recognition method according to an embodiment of the present disclosure.

When recognizing a location, posture information, which is a result of motion and posture estimation based on image data (spherical distance image, semantic segmentation image) obtained from sensor input according to the previously generated map data of each node and the above estimation method (spherical distance image, semantic segmentation image), and GPS information use

In an embodiment of the present disclosure, a pose graph expressing posture information as a graph based on a map graph is shown in FIG. 11 . 11 is an exemplary diagram illustrating a posture graph according to an embodiment of the present disclosure.

₁,

₂, …

₅,) 이다. 임의 노드에서 GPS 데이터가 입력될 수 있다. GPS 데이터가 있을 경우 GPS로부터 취득된 GPS 데이터(좌표 및 고도)를 에지로 갖는 GPS 노드(G₁, G₂)가 연결될 수 있다. 여기서 M₁, 및 M₂는 이미 생성된 지도 데이터의 노드를 나타낸다. Referring to FIG. 11 , in the attitude graph, edge information between nodes (x ₁ , x ₂ , ... x ₆ ) corresponding to the position of the vehicle at each point in time as the vehicle travels is the attitude information (

₁ ,

₂ , …

₅ ,). GPS data can be input from any node. When there is GPS data, the GPS nodes G ₁ , G ₂ having the GPS data (coordinates and altitude) obtained from the GPS as an edge may be connected. Here, M ₁ and M ₂ represent nodes of already generated map data.

Referring to FIG. 10 , as the vehicle drives, image data about the current point, that is, the current node, that is, a spherical distance image and a semantic segmentation image are acquired from data acquired from a sensor (LiDAR or camera, etc.) (S400). At this time, a spherical distance image and a semantic segmentation image are acquired based on the method described above (the method according to FIGS. 4 to 6 and the method according to FIGS. 7 and 8 ). Then, posture information is acquired using image data (spherical distance image and semantic segmentation image).

A sensor node, i.e., checks whether GPS data exists in the current node, and if there is GPS data, a GPS node (such a node is also called a unary factor) having coordinates and altitude obtained from the GPS data as an edge is used as the current node. connected to (S410). For example, as shown in FIG. 11 , if the current node is x ₁ , when GPS data exists, the GPS node G ₁ having the GPS data as an edge is connected to the current node x ₁ .

Then, an approximate current location is acquired based on the GPS data or the previous driving route (S420). Map data that has already been created based on the acquired location is searched for, and map data around the acquired location is found and stored in a temporary buffer (S430).

Thereafter, the posture information of the current node with respect to the searched map data is estimated ( S440 ). Here, motion and posture estimation may be performed based on the estimation method described above.

Then, the residual of the posture information estimated for the map data and the posture information according to the movement of the current node is compared with a threshold value (S450). For example, the average of the sum of the residuals is compared with a preset threshold, and when it is smaller than the threshold, the corresponding map data is connected from the graph to the map node (S460). For example, as in FIG. 11 , when the current node is x ₃ , map data corresponding to a case smaller than the threshold value is connected to the map node M ₁ and the current node x ₃ .

Thereafter, a precise location is obtained using GPS data and map data (S470). That is, as described above, the precise location is determined based on the map graph (eg, the posture graph of FIG. 11 ) in which the node connection is made. In this case, precise positioning can be made after optimization of the map graph is made.

Meanwhile, in step S430 , when map data does not exist, that is, when driving a new route, as in FIG. 11 , only a GPS node is added and optimized, such as node x ₁ and node x ₂ . .

Therefore, location recognition is performed by selectively using image data, GPS data, and map data on the optimized graph.

In the case of sensor nodes, although the error between nodes is very small, there is a problem that the error is accumulated and gradually increases. However, when constructing the graph as described above, it effectively suppresses the increase in sensor node error, making it possible to maintain global coordinates with high precision when generating new maps.

Next, a method for generating new map data will be described.

12 is a flowchart of a method for generating map data according to an embodiment of the present disclosure.

If map nodes are not connected during the location recognition process, or if the residual error is large even if they are connected, it is necessary to create or update map data. A point for generating new map data is preferentially set as an intersection area for loop closure detection. In areas other than intersections, map data is generated for each set range. However, the present disclosure is not necessarily limited thereto.

12, image data, ie, a spherical distance image and a semantic segmentation image, are acquired from data acquired from a sensor (LiDAR or camera, etc.) (S500). At this time, a spherical distance image and a semantic segmentation image are acquired based on the method described above (the method according to FIGS. 4 to 6 and the method according to FIGS. 7 and 8 ).

It is determined whether it is a map data generation point (eg, an intersection) based on the acquired image (S510). For example, in order to determine whether there is an intersection, a binary classifier using a deep neural network may be used. As an input of the binary discriminator, a spherical distance image and a semantic segmentation image can be used.

In areas other than the intersection, which is the map data generation point, it is determined to generate map data for each set range ( S520 ). For example, after the previous map data is generated, a point having the highest confidence score of the binary discriminator in the range of 0 to 60 m is selected as the map data generation point to generate new map data.

At the map data generation point, map data is synthesized to improve precision. To this end, first, coordinate transformation is performed on the data of the neighboring nodes of the node corresponding to the map data generation point (S530).

13 is an exemplary diagram illustrating map data generation according to an embodiment of the present disclosure.

For example, in FIG. 13A , when a node x ₃ is selected as a map data generation point, data of neighboring nodes is converted into coordinate data based on the node x ₃ and then synthesized. Coordinate transformation is performed based on edge information of each node.

Map data is synthesized based on the coordinate-transformed data. At this time, spherical distance images of each node (node x ₃ and its neighboring nodes) are synthesized, and semantic segmented images are synthesized.

을 갖는 3차원 메모리를 사용한다. 각 노드의 데이터는 에지 정보로부터 노드 x₃을 기준으로 하는 3차원 좌표로 변환하고, 구면 거리 영상 생성과 같은 방법으로 화소의 위치를 계산한 후(예를 들어, 구좌표계를 사용하여 2차원 표면에 투영하여 위치 획득), 계산된 위치에 대응하는 화소의 볼륨에 해당 값을 차례로 누적한다. 이렇게 볼륨 데이터의 초기화가 완료된다. 그다음, 도 13의 (c)에서와 같이, 각 개별 화소에 대입된 값들을 정렬(sort)하고, 예를 들어 설정 지점(예: 중간)에 위치한 값을 선택하여 2차원 영상 데이터를 생성한다. 여기서, 시간 중간값 필터(temporal median filter)를 사용하여 중간 위치의 값을 선택할 수 있다. 마지막으로, 도 13의 (d)에 예시된 바와 같이, 2차원 영상 데이터에서, 각 영상 화소

와 주변 화소 값을 함께 정렬하여 최대 값, 최소 값 및 중간 값을 구하고, 각 화소

값이 최대 값 또는 최소 값일 경우, 중간 값으로 대체하여 최종적으로 합성된 영상을 구한다 여기서, 적응 중간 값 필터(adaptive median filter)를 사용할 수 있다(S540). In the case of spherical distance image synthesis, as shown in (b) of FIG. 13, a volume of the same size as the image to be synthesized

3D memory with The data of each node is converted from edge information into three-dimensional coordinates based on the node x ₃ , and the position of the pixel is calculated in the same way as for generating a spherical distance image (e.g., a two-dimensional surface using a spherical coordinate system). ), and accumulates the corresponding values in the volume of the pixel corresponding to the calculated position in turn. In this way, initialization of volume data is completed. Then, as shown in (c) of FIG. 13 , the values substituted for each individual pixel are sorted and, for example, a value located at a set point (eg, the middle) is selected to generate 2D image data. Here, a value of an intermediate position may be selected using a temporal median filter. Finally, as illustrated in FIG. 13(d) , in the two-dimensional image data, each image pixel

and surrounding pixel values are sorted together to obtain the maximum, minimum, and median values, and each pixel

When the value is the maximum value or the minimum value, a final synthesized image is obtained by substituting an intermediate value. Here, an adaptive median filter may be used ( S540 ).

와 주변 화소 값을 함께 정렬하여 최대 값, 최소 값 및 중간 값을 구하고, 각 화소

값이 최대 값 또는 최소 값일 경우, 중간 값으로 대체하여 최종적으로 합성된 영상을 구한다.Meanwhile, even when synthesizing a semantic segmented image, volume data is initialized in the same way as spherical distance image synthesizing as described above. And instead of using the median filter, the 2D image data is analyzed by using a voting method that defines the value (the most distributed value) among the values assigned to the pixel as the representative value. create and each video pixel

and surrounding pixel values are sorted together to obtain the maximum, minimum, and median values, and each pixel

When the value is the maximum value or the minimum value, the final synthesized image is obtained by substituting the intermediate value.

Then, attribute information of the lane is acquired. In order for an autonomous vehicle to drive, not only location information but also attribute information of a lane to be driven is required. According to an embodiment of the present disclosure, attribute information of a vehicle is obtained by using the synthesized semantic segmentation image (S550). Since the synthesized semantic segmentation image classifies the lane, road boundary, and road surface marking information with high precision, it is possible to easily obtain the attribute information of the lane using this. For example, in the synthesized semantic segmentation image, lane and road boundary information is converted into line information by using Hough transform or Radon transform, etc. of image pixels, and 3D coordinates corresponding thereto are obtained. . In addition, road surface indication information such as speed limit and driving direction can be classified using existing classification techniques such as deep neural networks. Through this process, attribute information of the lane is acquired.

In this way, new map data including the synthesized spherical distance image, the synthesized semantic segmentation image, and lane attribute information are finally generated (S560).

Since synthesis is required only at the time of map data generation, it is desirable that the process of generating map data be operated in a separate process. It can be implemented very efficiently.

Meanwhile, for efficient storage and management of such map data, map data compression may be performed.

14 is an exemplary diagram illustrating compressed map data according to an embodiment of the present disclosure.

The map data includes, for each node of the graph, a spherical distance image, a semantic segmentation image, lane attribute information, and attitude information, and includes a time at which the map data is generated, GPS location information (GPS data), and the like. A spherical distance image and a semantic segmented image occupy most of the data storage space. Accordingly, these images are first compressed using a lossless image compression method such as PNG (Portable Network Graphics), JPEG-LS (Joint Photographic Experts Group-LosSless), JPEG2000, and FLIF (Free Lossless Image Format). do. For example, when the size of an image is 2048 and 192, respectively, it is reduced to about 200 kB through FLIF compression.

In addition, in order to further reduce the size of data and to manage it efficiently, the two primary compressed images are combined into one image and stored. To this end, data of a spherical distance image and data of a semantic segmented image are stored for each pixel.

As illustrated in the accompanying FIG. 14 ( a ), each pixel is set to have a set bit (eg, 16 bits) depth. The distance information of the spherical distance image is quantized in centimeter units and placed in the lower bit of the pixel (range data), and some information of the semantic segmented image is quantized and placed in the upper bit (semantic data) ).

In the case of distance information of a spherical distance image, sufficient precision can be obtained by having centimeter precision for a distance of up to 160 meters. In the semantic segmented image, only some information (segmentation information) such as lanes, road markings, and road boundaries is sampled, quantized, and placed in the upper bit of the pixel.

Since the data change amount of distance information is significantly higher than that of division information, higher compression efficiency can be expected by putting it in the lower bit. However, the present disclosure is not necessarily limited thereto.

As described above, the distance information of the spherical distance image and the segmentation information of the semantic segmented image, which are compressed and stored, can be separated into the original image through a simple bit operation. An example of a separated image is shown in (b) to (d) of FIG. 14 . 14(b) shows a spherical distance image separated from the stored pixel-specific information as shown in FIG. Represents a semantic segmented image. And Fig. 14(d) shows the final result of synthesizing Figs. 14(b) and 14(c).

It can be expected that the map data provided in the embodiment of the present disclosure is reduced to 1/100 or less compared to the existing three-dimensional coordinates. In an actual implementation, about 10 GB of 3D point data is generated for a 3D shape of 1 km, and storage space of about 500 MB or more is required when expressed as a voxel in units of 10 cm. However, according to an embodiment of the present disclosure, a size of 5 MB or less could be expressed with higher precision. For example, since the total road length in Korea is about 110,000 km, when using the method according to an embodiment of the present disclosure, it is possible to express a three-dimensional precision map of all roads in a size of about 500 GB, and the autonomous vehicle It is possible to mount a three-dimensional precision map of all roads.

The map data to be compressed in this way is registered and used. When the map data management part (refer to FIG. 2 ) receives the new map data, it may be compared with the existing map data to determine whether to register the new map data or update the existing map data. For example, when there is no existing map data in the coordinates around the received new map data, the new map data is registered. On the other hand, if there is existing map data, it is not updated immediately, but it is determined whether it is a temporary change or a measurement error, and the update can be selectively performed according to the result. To this end, if there is existing map data, new map data is registered as additional map data along with time information. Afterwards, when data collected from another device (eg, another vehicle) is received or when the map data of the corresponding point is repeatedly received by driving the same route, it is set by comparing it with the added map data registered with the time information. It is possible to update the existing map data based on the added map data only when different changes are maintained during the period. Comparison of map data may use residuals of posture estimation. In addition, it is desirable to limit the maximum number of additional map data, and to discard data that has not been used for a long time from time to time.

Meanwhile, optimization may be performed on the obtained map data according to an embodiment of the present disclosure.

15 is an exemplary diagram illustrating optimization of map data according to an embodiment of the present disclosure.

A map according to an embodiment of the present disclosure is a graph-based structure, and has a posture graph structure as shown in FIG. 15 . In this case, each node and edge of the graph is generated as shown in FIG. 15A. For a specific method, reference may be made to the method described above, and detailed description thereof will be omitted herein. Then, using the GPS location information, the GPS node (G) is connected to the graph. In this case, like node M ₃ , several GPS nodes may be connected using additional map data. A GPS node can be added if the error of the GPS information is large during repeated driving. Through this, it is possible to remarkably increase the positional accuracy.

The most important means to increase the precision in such a posture graph is to detect loop closures such as nodes M ₂ and M ₇ and optimize the graph after connecting the nodes. In the running of the vehicle, the roof closure usually occurs at the intersection point. As described above, a candidate node is selected using an intersection detection reliability score, a posture and a residual between the candidate nodes are calculated, and a loop closure is determined when the residual is less than or equal to a threshold value.

Finally, as shown in (b) of FIG. 15 , a high-resolution map of the lane level is generated by combining the attribute information of the lane included in each node using the optimized attitude information. Among the lane properties of each node, lane information is combined with the node's attitude information to convert it into lane information having global coordinates on the map, and compares it with neighboring nodes to find the intersection of the lanes. Using these intersections to connect lanes one after another, it is possible to create a lane-level high-resolution map that matches the optimized map. The lane-level map generated in this way is managed as a separate map layer, used for route planning of autonomous vehicles, etc., and can be updated together when the node link of the 3D map data is optimized.

16 is a diagram illustrating a structure of a location recognition apparatus according to an embodiment of the present disclosure.

16 , the location recognition device 1 includes an image data generator 11 that generates image data based on a sensor input, a motion and posture estimator 12 , and a location recognizer 13 . and further includes a map data generating unit 14 .

The image data generator 11 includes a spherical distance image generator configured to generate a spherical distance image by projecting a sensor input (three-dimensional coordinate information corresponding to three-dimensional spatial information) onto a two-dimensional plane, and a spherical distance image and corresponding thereto and a semantic segmentation image generator configured to generate a semantic segmentation image by performing semantic segmentation by receiving a texture image of The semantic segmentation image may be an image in which moving object information is removed and road vehicle information is included.

The motion and posture estimating unit 12 is configured to perform motion and posture estimation using a spherical distance image or a semantic segmented image, and to obtain the resultant posture information.

The location recognition unit 13 is configured to perform location recognition using already generated map data, posture information that is a result of motion and posture estimation based on image data (spherical distance image, semantic segmentation image), and GPS information.

The map data generating unit 14 is configured to generate map data including image data (spherical distance image, semantic segmentation image), posture information, and attribute information of a lane.

Since each of these components 11 to 14 is configured to implement the corresponding method described above, refer to the above description for specific functions.

The location recognition apparatus 1 according to an embodiment of the present disclosure may be implemented in a form mounted on a traveling vehicle, etc., and may operate in conjunction with the map data management apparatus 2 . For example, the location recognition device 1 may receive existing map data from the map data management device 2 or provide generated map data to the map data management device 2 . Meanwhile, the map data generator 14 is implemented in a form included in the location recognition device 1 , but is not limited thereto and may be implemented in a form included in other devices such as the map data management device 2 .

17 is a diagram illustrating a structure of an apparatus for managing map data according to an embodiment of the present disclosure.

17 , the map data management device 2 includes a map data registration processing unit 21 , a map data compression processing unit 22 , a data storage unit 23 , and a map data optimization processing unit 24 .

The map data registration processing unit 21 is configured to register map data as new map data or compare it with existing map data to update existing map data.

The map data compression processing unit 22 is configured to perform processing for storing the map data to be registered (simply referred to as storage processing). In particular, the map data compression processing unit 22 combines the image data constituting the map data, that is, a spherical distance image and a semantic segmented image into one image and processes it. For each pixel, the distance information of the spherical distance image is quantized and placed in the upper bit of the bit constituting the pixel, and the segmentation information of the semantic segmented image is sampled and quantized and placed in the lower bit of the bit constituting the pixel. Prior to such storage processing, the spherical distance image and the semantic segmented image may be first compressed, and the above storage processing may be performed on the compressed images.

The data storage unit 23 is configured to store map data. The map has a graph-based structure, and map data is stored for each node constituting the graph, and the map data includes a spherical distance image, a semantic segmentation image, lane property information, and posture information, and Includes time of creation, GPS location information, and more. The spherical distance image and the semantic segmentation image may be stored after being processed through the above storage process.

The map data optimization processing unit 24 is configured to perform optimization processing on graph-based map data.

Since each of these components 21 to 24 is configured to implement the corresponding method described above, refer to the above description for specific functions.

The map data management apparatus 2 according to an embodiment of the present disclosure may be implemented, for example, in the form of a server, and may operate in conjunction with the location recognition apparatus 1 . For example, the map data management device 2 receives map data newly generated from the location recognition device 1 and registers, stores, and optimizes the map data, or transfers the managed map data to the location recognition device 1 . can be provided as

18 is a structural diagram illustrating a computing device for implementing a method according to an embodiment of the present disclosure.

18 , the method (each method for map generation and management and location recognition described in the above embodiment) according to an embodiment of the present disclosure is implemented using the computing device 100 . can be

The computing device 100 may include at least one of a processor 110 , a memory 120 , an input interface device 130 , an output interface device 140 , a storage device 150 , and a network interface device 160 . . Each of the components may be connected by a bus 170 to communicate with each other. In addition, each of the components may be connected through an individual interface or a separate bus centering on the processor 110 instead of the common bus 170 .

The processor 110 may be implemented in various types such as an application processor (AP), a central processing unit (CPU), a graphic processing unit (GPU), and the like, and executes commands stored in the memory 120 or the storage device 150 . It may be any semiconductor device that does The processor 110 may execute a program command stored in at least one of the memory 120 and the storage device 150 . The processor 110 may be configured to implement the functions and methods described based on FIGS. 1 to 17 above. For example, the processor 110 may be implemented to perform the functions of an image data generator, a motion and posture estimator, a location recognizer, and a map data generator. In addition, the processor 110 may be implemented to perform the functions of a map data registration processing unit, a map data compression processing unit, and a data optimization processing unit.

The memory 120 and the storage device 150 may include various types of volatile or non-volatile storage media. For example, the memory may include a read-only memory (ROM) 121 and a random access memory (RAM) 122 . In an embodiment of the present disclosure, the memory 120 may be located inside or outside the processor 110 , and the memory 120 may be connected to the processor 110 through various known means.

The input interface device 130 is configured to provide data (sensor input) to the processor 110 , and the output interface device 140 is configured to output data (map, recognized vehicle location, etc.) from the processor 110 . do.

The network interface apparatus 160 may transmit or receive a signal with another device (eg, an autonomous driving system) through a wired network or a wireless network.

The input interface device 130 , the output interface device 140 , and the network interface device 160 may be collectively referred to as an “interface device”.

The computing device 100 having such a structure is named as a location recognition device/map data management device, and may implement the above methods according to an embodiment of the present disclosure.

In addition, at least some of the methods according to an embodiment of the present disclosure may be implemented as a program or software executed in the computing device 100 , and the program or software may be stored in a computer-readable medium.

In addition, at least some of the methods according to an embodiment of the present disclosure may be implemented as hardware capable of being electrically connected to the computing device 100 .

The embodiment of the present disclosure is not implemented only through the apparatus and/or method described above, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present disclosure, a recording medium in which the program is recorded, etc. Also, such an implementation can be easily implemented by those skilled in the art to which the present disclosure pertains from the description of the above-described embodiments.

Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improved forms of the present disclosure are also provided by those skilled in the art using the basic concept of the present disclosure as defined in the following claims. is within the scope of the right.

Claims (20)

A method of generating a map for autonomous driving, comprising:
obtaining 3D coordinate information corresponding to a 3D space;
obtaining a spherical distance image by projecting the three-dimensional coordinate information onto a two-dimensional plane;
generating a semantic segmentation image by performing semantic segmentation on the spherical distance image; and
Generating map data including the spherical distance image, the semantic segmentation image, and attribute information of the lane
How to include. According to claim 1,
The step of generating the semantic segmentation image is
performing semantic segmentation on the spherical distance image; and
generating the semantic segmented image including road information by removing a moving object from the image obtained by the semantic segmentation;
includes,
The road information is used as segmentation information including a pattern of a road surface, a road pattern, and a pattern around a road, and the attribute information of the road is obtained using the segmentation information included in the semantic segmentation image. According to claim 1,
The map has a graph-based structure, the map data is mapped for each node constituting the graph, and the edge information between the nodes includes posture information,
The method of claim 1, wherein the posture information is a result of motion and posture estimation using a gradient of the spherical distance image or a gradient of the semantic segmented image. 4. The method of claim 3,
The step of generating the map data is
acquiring map data of nodes surrounding the node in a state in which map data of a node corresponding to a map data generation point is generated; and
synthesizing the map data of the node and the map data of the neighboring nodes and using them as map data for the node
A method comprising 5. The method of claim 4,
The step of synthesizing the map data and using it as map data for the node comprises:
synthesizing the spherical distance image of the node and the spherical distance image of the neighboring node;
synthesizing the semantic segmented image of the node and the semantic segmented image of the neighboring node;
generating vehicle attribute information based on the synthesized semantic segmentation image; and
Using data including the synthesized spherical distance image, the synthesized semantic segmentation image, and the attribute information of the lane as map data for the node
A method comprising 6. The method of claim 5,
In the synthesizing of the spherical distance image, a synthesized spherical distance image is obtained based on a method of selecting an intermediate value from values accumulated for each pixel,
In the synthesizing of the semantic segmented image, a synthesized semantic segmented image is obtained based on a method of selecting a value having the largest distribution among values accumulated for each pixel. 7. The method of claim 6,
When synthesizing the spherical distance image and synthesizing the semantic segmented image,
coordinate transformation of the image of the node and the image of the neighboring node based on the node, and accumulating and storing the value of the corresponding pixel of the image of the node and the value of the corresponding pixel of the image of the neighboring node for each pixel; 2D image data is obtained by selecting one of the values accumulated for each pixel, and the final result of the corresponding pixel is obtained based on the result of comparing the values of the neighboring pixels with the values of the corresponding pixels for each pixel of the 2D image data wherein the synthesis is performed in a manner of selecting a value. According to claim 1,
compressing the map data
further comprising,
The compressing step
For each pixel, the spherical distance image and the semantic segmented image are combined into one image and stored, and for a pixel composed of a set bit, the distance information of the spherical distance image is placed in a first bit among the set bits, and the setting A method of locating segmentation information of the semantic segmentation image in a second bit among bits. 9. The method of claim 8,
The compressing step
compressing the spherical distance image and the semantic segmentation image;
quantizing the distance information of the compressed spherical distance image, and locating the quantized distance information in a higher bit among set bits of a pixel; and
Quantizing the segmentation information of the compressed semantic segmentation image, and locating the quantized segmentation information in a lower bit among set bits of a pixel
A method comprising According to claim 1,
registering the map data
further comprising,
The registration step is
registering the generated map data as new map data when there is no existing map data in the surrounding coordinates of the generated map data; and
updating the existing map data based on the generated map data when there is existing map data in the surrounding coordinates of the generated map data;
A method comprising 11. The method of claim 10,
The updating step is
repeatedly performing an operation of storing the generated map data as additional map data together with time information; and
comparing the stored additional map data and updating the existing map data based on the additional map data when different changes are maintained for a set time period;
A method comprising 4. The method of claim 3,
optimizing the graph constituting the map
further comprising,
In the optimizing step, when GPS data is obtained for each node, a GPS node having the GPS data is connected to the corresponding node, and when loop closure is detected in the graph, corresponding nodes are connected to each other, and each node using the posture information A method of combining attribute information of a lane included in a node. A method for recognizing a location based on a map for autonomous driving, comprising:
obtaining a spherical distance image based on data input from a sensor at the current node, and generating a semantic segmentation image from the spherical distance image;
acquiring an initial location based on GPS data or a previous driving route;
retrieving map data from the map based on the initial location; and
obtaining a current location based on the searched map data;
includes,
The map data of the map includes a spherical distance image, a semantic segmentation image, and attribute information of a lane. 14. The method of claim 13,
The map has a graph-based structure, the map data is mapped for each node constituting the graph, and the edge information between the nodes includes posture information,
The method of claim 1, wherein the posture information is a result of motion and posture estimation using a gradient of the spherical distance image or a gradient of the semantic segmented image. 15. The method of claim 14,
The step of obtaining the current location includes:
Estimate the posture information of the current node with respect to the searched map data, and when the estimated posture information and the posture information according to the movement of the current node satisfy a setting condition, the current position is obtained using the map data and, when there is GPS data in the current node, additionally using the GPS data to obtain the current location. A device for recognizing a location, comprising:
interface device; and
a processor coupled to the interface device and configured to perform location recognition
includes,
the processor is
obtaining a spherical distance image by projecting 3D coordinate information corresponding to a 3D space input through the interface device onto a 2D plane;
generating a semantic segmented image by performing semantic segmentation on the spherical distance image;
acquiring an initial location based on GPS data or a previous driving route;
retrieving map data from a map based on the initial location; and
Acquiring a current location based on the searched map data
, wherein the map data of the map includes a spherical distance image, a semantic segmentation image, and attribute information of a lane. 17. The method of claim 16,
The map has a graph-based structure, the map data is mapped for each node constituting the graph, and the edge information between the nodes includes posture information,
The apparatus of claim 1, wherein the posture information is a result of motion and posture estimation using a gradient of the spherical distance image or a gradient of the semantic segmented image. 17. The method of claim 16,
The processor further comprises:
Generating map data including the spherical distance image, the semantic segmentation image, and attribute information of a lane
is configured to perform
When the processor performs the operation of generating the map data,
acquiring map data of nodes surrounding the node in a state in which map data of a node corresponding to a map data generating point is generated;
synthesizing the spherical distance image of the node and the spherical distance image of the neighboring node;
synthesizing the semantic segmented image of the node and the semantic segmented image of the neighboring node;
generating vehicle attribute information based on the synthesized semantic segmentation image; and
Using data including the synthesized spherical distance image, the synthesized semantic segmentation image, and the attribute information of the lane as map data for the node
A device configured to do 17. The method of claim 16,
When the processor performs the operation of generating the semantic segmentation image,
performing semantic segmentation on the spherical distance image; and
An operation of generating the semantic segmentation image including road information by removing a moving object from the image obtained by the semantic segmentation
is configured to perform
The road information is used as segmentation information including a pattern of a road surface, a road pattern, and a pattern around a road, and the attribute information of the road is obtained using the segmentation information included in the semantic segmentation image. 17. The method of claim 16,
The map data is compressed and stored and managed, and when compressed, the spherical distance image and the semantic segmented image are combined into one image for each pixel and stored. The distance information of the spherical distance image is located, and the division information of the semantic segmentation image is located in a second bit of the setting bits.

Images