KR102275695B1 - Method And Apparatus for Real-Time Update of Three-Dimensional Distance Information for Use in 3D Map Building - Google Patents

Abstract

In constructing a 3D map, the present invention propagates 3D map information of an edge-point of a previous image frame to a current image frame, and provides an edge point depth ( Calculate SAD (Sum of Absolute Differences) cost according to edge-point depth), generate SAD cost volume using SAD cost, calculate optimal 3D distance information for edge point of current image frame, and Based on the existing 3D distance information corresponding to the edge point of the image frame and the current image frame, by applying a predetermined ratio to the optimal 3D distance information, the existing 3D distance information is corrected into new 3D distance information. Provided are an apparatus and method for real-time updating of 3D distance information, characterized in that it is updated.

Description

{Method And Apparatus for Real-Time Update of Three-Dimensional Distance Information for Use in 3D Map Building}

The present invention relates to an apparatus and method for real-time updating of 3D distance information in 3D map construction, and more particularly, by applying semi-dense SLAM (Simultaneous Localization And Mapping) to a current image frame, In constructing a 3D map that can be used to build a 3D map for an autonomous vehicle in real time and to update the current 3D map by calculating the cost volume of a plurality of previous image frames and the current image frame The present invention relates to an apparatus and method for real-time updating of 3D distance information.

In general, SLAM (Simultaneous Localization and Mapping) is a technology mainly used for autonomous driving such as mobile robots. It uses a map of the surrounding environment to recognize a location or vice versa to create a map of a specific area. , refers to a technology that performs both location recognition and map creation at the same time.

A mobile robot using SLAM generates a SLAM map by extracting key frames and landmarks from images acquired from a camera.

When the mobile robot moves in the area where the SLAM map is generated, it extracts keyframes and landmarks from the image acquired from the camera and compares the keyframes and landmarks of the previously generated map to recognize its location.

On the other hand, semi-dense SLAM is a method of using edge points as feature points instead of using feature points in the existing SLAM.

Semi-dense SLAM builds an overall 3-D environment map based on keyframes. Such semi-dense SLAM estimates the current 3D environment map (or 3D map) by obtaining an epipole line between the current frame and the current keyframe, and uses the estimated 3D environment map as the 3D environment of the keyframe. The map information is updated, and it is determined and selected whether the current frame is a new keyframe, and when it is determined as a new keyframe, the information of the previous keyframe is converted into information about the selected new keyframe and propagated.

In addition, a method of estimating 3D environment map information between a key frame and a current frame in semi-dense SLAM uses an Extended Kalman Filter (EKF) using probabilistic information or a simple filtering method. Accordingly, the conventional semi-dense SLAM makes it possible to construct an overall 3D environment map for all image frames.

However, in the method of constructing the environment map for all the image frames, it is difficult to know the 3D environment map in the current frame in real time, and an additional method is required to obtain the corresponding information. There is a problem in that information cannot be transmitted in real time. In addition, if a 3D environment map is built for every frame, there is a problem in that the 3D environment map update method for keyframes used in the conventional SLAM technology cannot be used.

Background art of the present invention is disclosed in Korean Patent Application Laid-Open No. 10-2016-0013713 (published on Feb. 5, 2016, apparatus and method for generating global routes for autonomous vehicles).

According to one aspect of the present invention, a three-dimensional map for an autonomous vehicle is built in real time by performing the semi-dense SLAM technology based on the current image frame, and the cost volume of a plurality of previous image frames and the current image frame is calculated. An object of the present invention is to provide an apparatus and method for real-time updating of 3D distance information in 3D map construction, which updates the current 3D map in real time by calculating it.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

According to an aspect of the present invention, in constructing a 3D map, there is provided a 3D map comprising: a propagator for propagating 3D map information of an edge-point of a previous image frame to a current image frame; a SAD cost calculator for calculating a sum of absolute differences (SAD) cost according to an edge-point depth between a predetermined number of previous image frames and a current image frame; a distance information calculator for generating an SAD cost volume using the SAD cost and calculating optimal 3D distance information for an edge point of the current image frame; Based on the existing 3D distance information corresponding to the edge point of the current image frame and the current image frame, the existing 3D distance information is corrected by applying a predetermined ratio to the optimal 3D distance information. There is provided a real-time updating device for 3D distance information, characterized in that it includes an updater for updating with new 3D distance information.

According to another aspect of the present invention, in constructing a 3D map, the method includes: a propagation process of propagating 3D map information of an edge-point of a previous image frame to a current image frame; a SAD cost calculation process of calculating a sum of absolute differences (SAD) cost according to an edge-point depth between a predetermined number of previous image frames and a current image frame; a three-dimensional distance information calculation process of generating an SAD cost volume using the SAD cost and calculating optimal three-dimensional distance information for an edge point of the current image frame; Based on the existing 3D distance information corresponding to the edge point of the current image frame and the current image frame, the existing 3D distance information is corrected by applying a predetermined ratio to the optimal 3D distance information. There is provided a real-time updating method of 3D distance information, characterized in that it includes an update process of updating to new 3D distance information.

According to one aspect of the present invention, the present invention enables semi-dense SLAM to be performed based on a current image frame, thereby enabling a three-dimensional map for an autonomous vehicle to be built in real time.

In addition, according to one aspect of the present invention, the current 3D map can be updated in real time by calculating the cost volumes of a plurality of previous image frames and the current image frame.

1 is an exemplary diagram showing a schematic configuration of a real-time three-dimensional map construction apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method for constructing a real-time 3D map according to an embodiment of the present invention.
3 is a flowchart illustrating a method of calculating a sum of absolute differences (SAD) cost volume in FIG. 2 .
4 is an exemplary diagram illustrating an SAD patch matching method according to an epipole line search in FIG. 3 .
5 is an exemplary diagram for explaining the meaning of the SAD cost volume in FIG. 3 .
6 is a graph illustrating an example of the relationship between the depth and the SAD cost for describing the SAD cost volume between the current frame and the previous frame in FIG. 3 .

Hereinafter, an embodiment of a real-time three-dimensional map construction apparatus and method according to the present invention will be described with reference to the accompanying drawings.

In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

According to an embodiment of the present invention, a 3D environment map (or 3D map) for a current image frame is generated in real time by using information about a previous image frame (eg, depth (ie, distance) information of edge points). It provides a way to build More specifically, an embodiment according to the present invention provides information on a previous image frame (eg, depth (distance) information of edge points) in order to apply semi-dense SLAM technology to a current image frame. ) to provide a method of obtaining a cost volume for the current image frame.

1 is an exemplary diagram showing a schematic configuration of a real-time three-dimensional map construction apparatus according to an embodiment of the present invention.

As shown in FIG. 1 , the real-time 3D map construction apparatus according to the present embodiment includes an image input unit 110 , an image processing unit 121 , and an edge and edge-point extraction unit 122 . , an edge-point tracing unit 123 , and a depth estimation unit 124 .

The image input unit 110 receives an image periodically photographed from a camera (not shown) mounted on the vehicle in frame units.

The image processing unit 121 binarizes the outline of the subject and the distribution of the image signal to facilitate edge extraction from each image frame (or frame) periodically input from the image input unit 110 .

The edge and edge point extraction unit 120 extracts an edge from each image frame processed by the image processing unit 120, and a plurality of edge points (eg, semi-dense candidates) are selected from the extracted edge. edge points).

The extracted edges and edge points are extracted for each periodically input image frame, and the edge point tracking unit 123 tracks a plurality of edge points whenever an image frame is newly input.

The depth estimator 124 estimates optimal depths for the edge points tracked by the edge point tracking unit 140, respectively, and reflects the estimated optimal depths on an internally generated 3D map (not shown). to update In this case, the depth may be a distance between edge points. In addition, the optimal depth may be a depth having a minimum sum of absolute differences (SAD) cost.

The operations of the image processing unit 121 , the edge and edge point extraction unit 122 , the edge point tracking unit 123 , and the depth estimation unit 124 may be integrated into one control unit 120 to function. The controller 120 may be implemented as a microprocessor (not shown) operating in a programmable manner by software.

2 is a flowchart illustrating a method for constructing a real-time 3D map according to an embodiment of the present invention.

A method of constructing a 3D environment map (or 3D map) for a current image frame in real time using information (eg, depths of edge points) on a previous image frame will be described with reference to FIG. 2 . do.

The controller 120 propagates the 3D environment map information of the edge points of the previous image frame to the current image frame (S101).

In this case, the propagation is the edge point reflected in the 3D map so that the controller 120 reflects the depths of the edge points of the previous image frame on the 3D map, and calculates the SAD cost by comparing the previous image frame with the current image frame. This can be achieved by projecting the

The controller 120 calculates a sum of absolute differences (SAD) cost according to depths of edge points between a predetermined number of previous image frames and the current image frame ( S102 ).

The SAD cost corresponds to the moving direction and moving distance of the camera mounted on the vehicle, and the patch for the edge points of each image (eg, the previous image frame and the current image frame, etc.) is constant around the edge point. The value of the specified rectangular window area) means the same degree. For example, the SAD cost value decreases in proportion to the difference in the patch value. Therefore, if the SAD cost value is the minimum, it may be determined that the optimal depths for the edge points are estimated.

The controller 120 generates an SAD cost volume using the SAD costs, and calculates new optimal 3D distance information for edge points of the current image frame (S103).

The control unit 120 applies a predetermined ratio to the existing 3D distance information for the edge points, that is, 3D distance information based on previous image frames and 3D distance information newly obtained based on the current image frame. It is updated with new 3D distance information by performing correction (S104). For example, the ratio of the existing 3D distance information and the newly obtained 3D distance information may be reflected as 5:5, or the ratio of the existing 3D distance information and the newly obtained 3D distance information may be reflected as 8:2.

An embodiment according to the present invention makes it possible to build a 3D environment map for a current image frame instead of the conventional method of constructing a keyframe-based 3D environment map, and thus, unlike the prior art, a previous image frame A method of constructing a 3D environment map by calculating the SAD cost volume using the information of

Hereinafter, a method of calculating a SAD cost volume according to a depth between a predetermined number of previous image frames and a current image frame will be described in detail with reference to FIG. 3 .

3 is a flowchart illustrating a method of calculating a sum of absolute differences (SAD) cost volume in FIG. 2 .

The control unit 120 controls an epipole line for each edge point of edges designated as semi-dense candidates in each image frame (ie, previous image frames and current image frame). ) is calculated (S201).

In this case, the epipole line is information expressed through epipole matching. Epipole matching is a method of finding two consecutive image frames. When each camera position information is known for the previous image frame and the current image frame, one edge point of the previous image is converted to one epipole line from the current image frame. It is expressed and searches for the most similar point on the epipole line (eg, the edge point with the minimum SAD cost value).

After calculating the epipole lines of the respective edge points, the controller 120 samples the epipole lines at a plurality of predetermined depths to re-project the current image frame to the previous image frame (S202). ).

The re-projection has the opposite meaning to the propagation described in FIG. 2 , and the controller 120 reflects the depths of the edge points of the current image frame on the 3D map and compares the previous image frame with the current image frame. It means an operation of projecting the edge point reflected in the 3D map onto the previous image frame for calculating the SAD cost through the

The controller 120 calculates the SAD cost by comparing the re-projected edge point on the previous image frame with the patch of the edge point on the current image frame (S203, see FIG. 4 ).

The controller 120 calculates the SAD cost through patch comparison by performing re-projection on several image frames (step S102), and calculates the SAD cost volume using the calculated SAD cost value (S204).

The control unit 120 calculates an optimal depth value from the SAD cost volume and applies it to the 3D environment map (S205, see FIG. 6 ).

FIG. 4 is an exemplary diagram illustrating an SAD patch matching method according to an epipole line search in FIG. 3 .

5 is an exemplary diagram for explaining the meaning of the SAD cost volume in FIG. 3 .

FIG. 6 is a graph illustrating a relationship between a depth and an SAD cost for describing an SAD cost volume between a current frame and a previous frame in FIG. 3 .

Referring to FIG. 4 , the epipole line is usually shown in the form of a straight line in a pin-hole camera, but appears in the form of a curved line in a wide-angle camera. Therefore, the distance range of the 3D environment map to be constructed is determined, and disparity at regular intervals is sampled and compared while searching on the epipole line. In this case, the search is performed by extracting a patch of a certain size from each point and then matching the sampling point (ie, edge point) with the smallest SAD cost value using the SAD cost.

Meanwhile, as shown in FIG. 5 , the cost volume shows a tendency and a change process of disparity of any one edge point with respect to a plurality of previous image frames. In other words, a number of disparity values for previous image frames can be known through the SAD cost volume, and this automatically occurs when the corresponding edge point is invisible due to being obscured by an obstacle among the previous image frames, and when an error occurs, the error to avoid being included in the calculation.

Referring to FIG. 6 , as a result of sampling an edge point having a true depth of 22.3074 m by applying the method according to the present embodiment, it can be seen that a depth value (about 23 m) that is almost close to the actual depth is calculated. have. Each graph of FIG. 6 represents an SAD cost corresponding to one edge point for each frame. The SAD cost volume is generated by accumulating these SAD costs, and the depth with the largest minimum value of the SAD cost is applied to the 3D environment map.

As described above, according to an embodiment of the present invention, a 3D environment map can be constructed for the current image frame in real time, and optimal 3D environment map information can be obtained using the SAD cost volume between previous image frames and the current image frame. has the effect of obtaining

As described above, the present invention has been described with reference to the embodiment shown in the drawings, but this is merely exemplary, and various modifications and equivalent other embodiments are possible therefrom by those of ordinary skill in the art. will understand the point. Therefore, the technical protection scope of the present invention should be defined by the following claims.

110: video input unit
120: control unit
121: image processing unit
122: edge and edge point extraction unit
123: edge point tracking unit
124: depth estimation unit

Claims (9)

In constructing a 3D map,
a propagator for propagating 3D map information of an edge-point of a previous image frame to a current image frame;
a SAD cost calculator for calculating a sum of absolute differences (SAD) cost according to an edge-point depth between a predetermined number of previous image frames and a current image frame;
a distance information calculator for generating a SAD cost volume using the SAD cost and calculating current 3D distance information, which is 3D distance information corresponding to an edge point of the current image frame;
By applying a predetermined ratio between the existing 3D distance information corresponding to the edge point of the current image frame and the current 3D distance information, the existing 3D distance information is corrected and updated to new 3D distance information ( update)
3D distance information real-time update device comprising a. The method of claim 1,
The edge point is
A real-time updating device for 3D distance information, characterized in that it is extracted for each periodically input image frame. The method of claim 1,
The SAD cost is
Corresponding to the moving direction and moving distance of the camera mounted on the vehicle, by comparing patches for the edge point of the previous image frame and the edge point of the current image frame, the patch indicates the same degree Real-time update device of 3D distance information The method of claim 1,
and a depth processor estimating a depth having the minimum SAD cost as an optimal depth and applying the estimated optimal depth to the 3D map. The method of claim 1,
The SAD cost volume is
An epipole line is calculated for each edge point of edges designated as candidates for a semi-dense method among the previous image frame and the current image frame, and the epipole line of each edge point is calculated After each calculation, the epipole lines are sampled at a plurality of predetermined depths and re-projected from the current image frame to the previous image frame. Real-time update device of information. 6. The method of claim 5,
The SAD cost calculator,
and calculating the SAD cost by comparing each edge point of the re-projected previous image frame with a patch of each edge point of the current image frame. 6. The method of claim 5,
and a depth value estimator for estimating a depth value between the edge points of the current image frame from the SAD cost volume and applying it to a 3D environment map. In constructing a 3D map,
a propagation process of propagating 3D map information of an edge-point of a previous image frame to a current image frame;
a SAD cost calculation process of calculating a sum of absolute differences (SAD) cost according to an edge-point depth between a predetermined number of previous image frames and a current image frame;
a three-dimensional distance information calculation process of generating a SAD cost volume using the SAD cost and calculating current three-dimensional distance information, which is three-dimensional distance information corresponding to an edge point of the current image frame;
By applying a predetermined ratio between the existing 3D distance information corresponding to the edge point of the current image frame and the current 3D distance information, the existing 3D distance information is corrected and updated with new 3D distance information update process
3D distance information real-time update method comprising a. 9. The method of claim 8,
The method of claim 1, further comprising: a depth application process of estimating a depth having the minimum SAD cost as an optimal depth and applying the estimated optimal depth to the 3D map.

Images