KR102336523B1 - Mobile mapping system that improves map information using point cloud data - Google Patents

Abstract

In accordance with the present invention, provided is a map production system capable of efficiently improving map information by performing up-sampling on point group data included in the map information and applying color information. In accordance with one embodiment, the map production system includes: a sensor part including a lidar sensor or a radar sensor and an inertia sensor and provided to acquire position information corresponding to integrated map information; a camera provided to acquire image information of the integrated map information; a memory storing the integrated map information; and a processor forming the integrated map information using the position information and the image information, performing a predetermined first operation on the integrated map information to separate the integrated map information into ground surface map information and on-ground map information, converting the integrated map information into a lidar coordinate system through a predetermined second operation, converting the lidar coordinate system into an image coordinate system through a predetermined third operation, extracting color information corresponding to each image coordinate system, altering the ground surface map information using the color information, and merging the altered ground surface map information with the on-ground map information to form the integrated map information.

Description

A map production system that improves map information using point cloud data {MOBILE MAPPING SYSTEM THAT IMPROVES MAP INFORMATION USING POINT CLOUD DATA}

The present invention relates to a technology for improving map information used for autonomous driving, and more particularly, to a technology related to a mobile mapping system for up-sampling point cloud data constituting map information.

A mobile mapping system (MMS) refers to a device that collects geospatial data equipped with various remote sensing systems such as cameras and lidar, and navigation sensors such as GPS (Global Positioning System) and INS (Inertial Navigation System). .

A precise point cloud map can be built using data collected from MMS, including LiDAR.

It is possible to add color information to a 3D point cloud map using the RGB image data collected by the camera.

This precision point cloud map can be implemented as a 3D three-dimensional map with superior precision compared to the general map for hazards, and can include various information on the road environment such as lanes, road markings, crosswalks, and traffic signs.

On the other hand, to form such a precise point cloud map, data corresponding to the ground may be insufficient. In particular, when color information is insufficient in the corresponding area, map information and RGB data are matched, so it is difficult to add color information, and research to solve this problem is being actively conducted.

[Prior art literature]

Laid-open Patent Publication KR 10-2019-0043035 (2019.04.25)

The present invention provides a cartography system for efficiently improving map information by performing upsampling of point cloud data included in map information and applying color information.

A map production system according to an embodiment includes a sensor unit including a lidar sensor or a radar sensor and an inertial sensor and configured to acquire location information corresponding to integrated map information;

a camera provided to acquire image information of the integrated map information; a memory for storing the integrated map information; and

Forming the integrated map information using the location information and the image information, performing a first predetermined operation on the integrated map information to separate the integrated map information into ground map information and ground map information,

The ground map information is converted into a lidar coordinate system by performing a second predetermined operation,

The lidar coordinate system is converted into an image coordinate system by performing a third predetermined operation,

extracting color information corresponding to each of the image coordinate systems;

and a processor that changes the ground map information by using the color information, and forms the integrated map information by merging the changed ground map information and the ground map information.

In the first operation, a flip operation is performed on each of the height data included in the integrated map information, and the ground map information is formed using a set of a plurality of planes on the height data on which the flip operation is performed. .

In the first operation, a predetermined interpolation algorithm (Moving Least Square) is performed on a region corresponding to the plurality of planes to form the ground map information.

The image processor,

Up-sampling of data of the ground map information may be performed based on the first operation.

The second operation may convert the Cartesian coordinates of the map information into the LiDAR coordinate system based on a coordinate transformation matrix and a rotation transformation matrix between the lidar sensor and the inertial sensor.

The second operation is an inverse operation of a matrix that converts the lidar sensor coordinates into the inertial sensor coordinates, an inverse operation of a matrix that converts GPS coordinates into rotational coordinates, and an inverse operation of a matrix that converts the inertial coordinate system into a map information coordinate system and a calculation of multiplying the difference between the map information and the GPS information,

It may include an operation of converting the Cartesian coordinates of the map information into the LiDAR coordinate system based on a coordinate transformation matrix and a rotation transformation matrix between the lidar sensor and the inertial sensor.

The second operation is

The orthogonal coordinates of the map information may be converted into the lidar coordinate system based on a difference between the map information and the GPS information corresponding to each of the map information.

The third operation may include a matrix formed based on a focal length and a main point of the camera that has acquired an image based on the formation of the integrated map information.

The third operation may convert the lidar coordinate system into the image coordinate system based on a coordinate transformation matrix and a rotation transformation matrix between the lidar sensor or the radar sensor and the camera.

The image processor,

matching each of the image coordinates with the ground map information, and changing the ground map information using the color information corresponding to the image coordinates,

The resolution of the integrated map information may be improved based on merging the changed ground map information and the ground map information.

The image processor,

Forming the integrated map information including height information corresponding to the surrounding object obtained through the lidar sensor,

The integrated map information may be divided into the ground map information and the ground map information based on the height information.

The map production system according to an exemplary embodiment may perform upsampling of point data included in the map information and apply color information to efficiently improve the resolution of the map information.

The cartography system according to an embodiment may change the sparse point distribution of the road surface to a high density.

In the cartography system according to an embodiment, the cartography system can more efficiently register lanes or road surface marks on map information.

The map production system according to an embodiment may provide improved map information to provide safe and efficient autonomous driving.

1 is a control block diagram of a cartography system according to an embodiment.
2 is a flowchart illustrating an operation of the present invention according to an embodiment.
3 is a diagram for explaining an operation of dividing integrated map information into ground map information and ground map information according to an embodiment.
4A and 4B are diagrams for explaining an operation of performing up-sampling on points included in map information according to an exemplary embodiment.
5A and 5B are diagrams illustrating an improved resolution of map information according to an exemplary embodiment.

Like reference numerals refer to like elements throughout. This specification does not describe all elements of the embodiments, and general content in the technical field to which the present invention pertains or content that overlaps between the embodiments is omitted. The term 'part, module, member, block' used in this specification may be implemented in software or hardware, and according to embodiments, a plurality of 'part, module, member, block' may be implemented as one component, It is also possible for one 'part, module, member, block' to include a plurality of components.

Throughout the specification, when a part is "connected" to another part, it includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Throughout the specification, when a member is said to be located "on" another member, this includes not only a case in which a member is in contact with another member but also a case in which another member exists between the two members.

Terms such as first, second, etc. are used to distinguish one component from another, and the component is not limited by the above-mentioned terms.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In each step, the identification code is used for convenience of description, and the identification code does not describe the order of each step, and each step may be performed differently from the specified order unless the specific order is clearly stated in the context. have.

1 is a control block diagram of a cartography system 1 according to an embodiment.

The map production system 1 according to an embodiment may include a sensor unit 100 , a camera 200 , a memory 400 , and a processor 200 .

The sensor unit 100 may include a lidar sensor, a radar sensor, and an inertial sensor.

Based on the sensor provided in the sensor unit 100 , location information of a surrounding object required for autonomous driving of the vehicle may be acquired.

The radar sensor may refer to a sensor that emits electromagnetic waves of about microwave (microwave, 10 cm to 100 cm wavelength) to an object, receives electromagnetic waves reflected from the object, and detects a distance, direction, altitude, etc. from the object.

The lidar sensor may refer to a sensor that precisely draws a surrounding state by emitting a laser pulse, receiving the light reflected from a surrounding target object, and measuring the distance to the object.

The inertial sensor may mean a sensor including a gyro sensor and an acceleration sensor.

The camera 200 may be provided in a configuration to acquire surrounding image information.

According to an embodiment, the camera 200 may be provided at the front, rear, and side of the vehicle to acquire an image.

The camera 200 may include a charge-coupled device (CCD) camera 230 or a CMOS color image sensor.

The memory 400 may store integrated map information.

The integrated map information may refer to map information used for autonomous driving.

The integrated map information may include ground map information corresponding to the ground information and ground map information excluding the ground map information from the integrated map information.

The memory 400 is a non-volatile memory device or RAM (such as a cache, read only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) and flash memory (Flash memory). It may be implemented as at least one of a volatile memory device such as a random access memory, a hard disk drive (HDD), or a storage medium such as a CD-ROM, but is not limited thereto.

The processor 200 may divide the integrated map information into ground map information and ground map information by performing a first predetermined operation on the integrated map information.

The first operation may refer to an operation including a Cloth Simulation Filter (CSF) surface separation algorithm and an interpolation algorithm (Moving Least Square), that is, an up-sampling algorithm, as will be described later.

The processor 200 may convert the ground map information into a LIDAR coordinate system by performing a second predetermined operation.

The second operation may refer to an operation of converting the world coordinate system constituting the map information into the lidar coordinate system.

The second operation for converting the world coordinate system into the LIDAR coordinate system may include an operation for converting the Cartesian coordinate system into the rotational coordinate system, and a detailed description related thereto will be described below.

The processor 200 may convert the LIDAR coordinate system into an image coordinate system by performing a third predetermined operation.

The third operation may include a matrix determined based on the focal length and main point of the camera that acquired the image.

The processor 200 may extract color information corresponding to each image coordinate system.

The processor 200 may change ground map information by using color information, and may merge the changed ground map information and ground map information.

The processor 200 may form integrated map information based on this operation.

The processor 200 may perform a flip operation on each of the height data included in the integrated map information through the first operation described above.

The processor 200 may form the ground map information by using a set of a plurality of planes on the height data on which the flip operation is performed.

The flip operation may refer to an operation of flipping each height information included in the map information.

A plurality of planes on the height data on which the flip operation is performed may mean planes formed in a flipped state, and a detailed description thereof will be described later.

The processor 200 may refer to an operation of forming ground map information by performing a predetermined interpolation algorithm (Moving Least Square) on a region corresponding to a plurality of planes through the above-described first operation.

A predetermined interpolation algorithm may refer to an operation of predicting a point without data when data of two points, which are usually obvious to a person skilled in the art, is known.

In the portion where the height information is not continuous through the flip operation, the processor 200 may acquire the height information of the corresponding region through the interpolation algorithm included in the first operation.

The processor 200 may perform up-sampling of data of ground map information.

The processor 200 may perform a second operation to convert the Cartesian coordinates of the map information into the LiDAR coordinate system based on the coordinate transformation matrix and the rotation transformation matrix between the lidar and the inertial sensor provided in the vehicle.

The processor 200 may perform a second operation to convert the Cartesian coordinates of the map information into the LIDAR coordinate system based on a difference between the map information and the GPS information corresponding to each of the map information.

A detailed description of the above-described second operation will be provided below.

The above-described third operation may include a matrix formed based on a focal length and a main point of a camera that has acquired an image based on the formation of the integrated map information.

The processor 200 may perform a third operation to convert the lidar coordinate system into an image coordinate system based on the coordinate transformation matrix and the rotation transformation matrix between the lidar and the camera.

A detailed description of the matrix and the like included in the third operation will be described below.

Meanwhile, the processor 200 may match each image coordinate with ground map information.

Also, the ground map information can be changed by using color information corresponding to the image coordinates.

The processor 200 may improve the resolution of the integrated map information based on the changed ground map information and the merging of the ground map information.

Meanwhile, when a color is added to the up-sampled map information, the sharpness of the map information is increased, and thus the resolution may be increased.

The processor 200 may form the integrated map information including height information corresponding to surrounding objects obtained through the lidar sensor, and divide the integrated map information into ground map information and ground map information based on the height information. have.

That is, the lidar sensor may acquire height information of the object from the sensor.

When the height of the object obtained by the lidar sensor exceeds a predetermined height, the image processor may determine that the object is information corresponding to ground map information, not an area corresponding to the ground.

The processor includes a memory (not shown) that stores data for an algorithm or a program reproducing the algorithm for controlling the operation of components in the mobile mapping system, and a processor that performs the above-described operation using the data stored in the memory ( not shown) may be included. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

At least one component may be added or deleted according to the performance of the components of the mobile mapping system shown in FIG. 1 . In addition, it will be readily understood by those of ordinary skill in the art that the mutual positions of the components may be changed corresponding to the performance or structure of the system.

Meanwhile, each component illustrated in FIG. 1 refers to software and/or hardware components such as Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC).

2 and 3 are flowcharts illustrating the operation of the present invention according to an embodiment.

Referring to FIG. 2 together, the image processor may separate the integrated ground map information into map information and ground map information (S201).

In separating them, the first operation shown in FIGS. 4A and 4B may be performed as will be described later.

In addition, the image processor may convert the map information of the ground into a lidar coordinate system (S202). Meanwhile, for the corresponding operation, the image processor may perform a second operation.

The second operation may be performed based on Equation 1 below.

Referring to Equation 1, L may mean a lidar coordinate system, and W may mean a coordinate corresponding to map information. G may mean GPS data.

In addition, R _L2I may mean a matrix converting the rotation coordinate system of the _{lidar into an inertial coordinate system, and R ckpt} may mean a matrix converting GPS coordinates into rotation coordinates.

Also, R _I2W may mean a matrix that transforms the inertial coordinate system into the map information coordinate system. T _L2I may mean a determinant for converting the LIDAR rectangular coordinate system into the inertial coordinate system.

The image processor may convert the coordinates constituting the map information into lidar coordinates using Equation 1 above.

Also, the image processor may convert the lidar coordinate system into an image coordinate system (S203).

When the processor converts the LIDAR coordinate system into the image coordinate system, a third operation based on Equation 2 below may be performed.

Referring to Equation 2, Pixel _x , Pixel _y means image coordinates, and f _x and f _y may mean a focal length of a lens that acquires an image. c _y and c _x may refer to a main point of a lens for acquiring an image. The main point may mean a conjugate point having a horizontal magnification of +1 on the axis of the coaxial optical system.

It may also mean a matrix that changes the rotation coordinates of the _{R L2C lidar to the image rotation coordinate system.} In addition, T _L2C may mean a matrix that changes the Cartesian coordinates of the lidar to the image Cartesian coordinate system. L may mean a coordinate system of the lidar.

Meanwhile, the above-described operation Equation 2 may mean a perspective projection transformation.

On the other hand, when converted to an image coordinate system based on the above-described operation, the image processor may extract color information corresponding to each image coordinate system and change the color of the map information using the extracted color information (S204).

That is, the processor may convert data in the LIDAR coordinate system into the image coordinate system.

As a result, the processor can project the points in the map information to the image coordinate system to finally obtain the position of the image coordinates corresponding to the map information.

When the processor reads the RGB values on the image coordinates, it can add color values to the data on the map information.

In addition, the processor may generate integrated map information by merging the ground map information and the ground map information (S205).

That is, the processor may perform up-sampling of the point cloud data of the map information and merge the color information-coated ground map and the ground map information to form the finally determined integrated map information.

Meanwhile, referring to FIG. 3 , it is a flowchart illustrating an operation in which the image processor performs upsampling.

Referring to FIGS. 3 and 4A and 4B together, the image processor may perform a flip operation of ground map information ( S301 ).

Meanwhile, referring to FIG. 4A together, it may mean PD41 integrated map information.

Referring to FIGS. 4A and 4B together, when a flip operation is performed on the height information of the PD41, the height inverse operation is performed to convert the height information like the PD42.

Meanwhile, the image processor may determine a flat set of height data of the ground map information (S302).

That is, the data of the map information converted based on the above-described operation may form a plane (S5).

The processor may perform upsampling only on road surface data provided on the map information. The algorithm for the processor to perform the above-described operation may be implemented as a Cloth Simulation Filter (CSF).

As shown in FIG. 4B , the processor may use a method of separating the corresponding plane by extracting only points that are in contact with the cloth, assuming that the map information is turned over and a cloth is placed thereon.

The processor may perform upsampling using an interpolation algorithm between planes (S303).

Referring again to FIG. 4B , the image processor may form a ground surface in the V4 region where the space is empty by using a plane interpolation method.

Finally, the image processor may extract the area corresponding to S4 as ground map information and separate the integrated map information into ground map information and the rest of the ground map information.

5A and 5B are diagrams for explaining an operation of performing up-sampling on points included in map information according to an exemplary embodiment.

Based on the above-described operation, if the processor projects color information on the map information, the sparse point cloud data may be densely formed on the existing map information.

Based on such an operation, the map production system may more easily perform a lane or road surface marking registration operation of map information.

For example, when visualizing road data collected with an aerial view as shown in FIG. 5A , lanes or road markings (unprotected left turns) are not well revealed.

That is, when map information is formed based on previously acquired information as shown in FIG. 5A , the density of point cloud data is low and it is difficult to grasp even a schematic shape.

Therefore, in this case, it is difficult to form accurate map information.

Specifically, in the case of the M5A region shown in FIG. 5A, it is difficult to determine which type of information is the same marker as only the extent to which a certain shape exists on the road can be predicted.

In this area, the density of the point cloud data is insufficient, and thus information may be difficult to grasp despite the presence of the point cloud data.

Based on the above-described operation, the map production system may perform upsampling of the point cloud data of the map information and project the color information to form map information in the form of M5B.

The map information of the existing M5A was difficult to grasp the shape of, but through the above-described operation, it can be converted into map information including a clear shape of “unprotected left turn”.

In summary, if the corresponding map information is converted into the LiDAR coordinate system based on the above operation and the corresponding color information is matched by converting it into the image coordinate system, as shown in FIG. .

6A and 6B are diagrams illustrating an improved resolution of map information according to an exemplary embodiment.

Referring to FIG. 6A , the map information D61 is shown before upsampling is performed, and FIG. 6B is shown after performing upsampling on the map information D62 based on the above-described operation.

The cartography system may finally form upsampled map information by merging the ground map information to which the color information is applied by upsampling and the ground map information (D62).

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create a program module to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

The computer-readable recording medium includes any type of recording medium in which instructions readable by the computer are stored. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like.

The disclosed embodiments have been described with reference to the accompanying drawings as described above. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be practiced in other forms than the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

1: Cartography system
100: sensor unit
200 : processor
300 : camera
400: storage

Claims (10)

a sensor unit including a lidar sensor or a radar sensor and an inertial sensor and provided to acquire location information corresponding to integrated map information;
a camera provided to acquire image information of the integrated map information;
a memory for storing the integrated map information; and
forming the integrated map information using the location information and the image information;
performing a first predetermined operation on the integrated map information including the ground map information and the ground map information;
The ground map information is converted into a lidar coordinate system by performing a second predetermined operation,
The lidar coordinate system is converted into an image coordinate system by performing a third predetermined operation,
extracting color information corresponding to each of the image coordinate systems;
a processor for changing the ground map information using the color information, and merging the changed ground map information and the ground map information to form the integrated map information; According to claim 1,
The first operation is
performing a flip operation on each of the height data included in the integrated map information,
A map production system for forming the ground map information by using a set of a plurality of planes on the height data on which the flip operation has been performed
3. The method of claim 2,
The first operation is
A map production system for forming the ground map information by performing a predetermined interpolation algorithm (Moving Least Square) on a region corresponding to the plurality of planes
According to claim 1,
The processor is
A map production system for performing up-sampling of data of the ground map information based on the first operation
According to claim 1,
The second operation is
Inverse operation of the matrix for converting the LIDAR sensor coordinates into the inertial sensor coordinates, the inverse operation of the matrix converting the GPS coordinates into rotational coordinates, the inverse operation of the matrix for converting the inertial coordinate system into the map information coordinate system, and the map information and the Including the operation of multiplying the difference of GPS information,
A map production system comprising an operation for converting the Cartesian coordinates of the map information into the LiDAR coordinate system based on a coordinate transformation matrix and a rotation transformation matrix between the lidar sensor and the inertial sensor
6. The method of claim 5,
The second operation is
A map making system that converts orthogonal coordinates of the map information into the LIDAR coordinate system based on a difference between the map information and the GPS information corresponding to each of the map information
According to claim 1,
The third operation is
A map production system including a matrix formed based on a focal length and a main point of the camera that has acquired an image based on the formation of the integrated map information
8. The method of claim 7,
The third operation is
A map making system that converts the lidar coordinate system into the image coordinate system based on a coordinate transformation matrix and a rotation transformation matrix between the lidar sensor or the radar sensor and the camera
According to claim 1,
The processor is
Match each of the image coordinates with the ground map information,
changing the ground map information by using the color information corresponding to the image coordinates,
A map production system for improving the resolution of the integrated map information based on merging the changed ground map information and the ground map information
According to claim 1,
The processor is
Forming the integrated map information including height information corresponding to the surrounding object obtained through the lidar sensor,
A map production system for separating the integrated map information into the ground map information and the ground map information based on the height information

Images