KR101957089B1 - Method and system on deep self-guided cost aggregation for stereo matching - Google Patents

Abstract

According to an embodiment of the present invention, a method for aggregating deep magnetic induction costs for stereo matching can obtain an accurate disparity map. To this end, the method for aggregating deep magnetic induction costs for stereo matching comprises the steps of: generating a cost volume slice from a pair of stereo images; matching the stereo images based on the generated cost volume slice; and aggregating magnetic induction matching costs by matching the stereo images to obtain a disparity map.

Description

[0001] METHOD AND SYSTEM FOR DEEP SELF-GUIDED COST AGGREGATION FOR STEREO MATCHING [0002]

The following description relates to a magnetic induction cost aggregation technique for stereo matching.

Human vision is one of the senses for acquiring information of the surrounding environment, and it is possible to recognize the location and distance of objects through the two eyes. The visual system of the robot consists of a stereo camera (that is, a left camera and a right camera), and reads images from two cameras to obtain a single And reconstructed into three-dimensional information. In this case, one image means that a three-dimensional space is projected into a two-dimensional space. In this process, the three-dimensional distance information (depth) is lost, and it is very difficult to directly restore the three-dimensional space. However, if there are two or more images obtained from different positions, the restoration of the three-dimensional space can be performed. In other words, when a point on the real space is formed on two images, the corresponding point of the point appearing in the two images can be found, and the geometrical structure can be used to find the position in the real space of the point. Finding the correspondence of two images (hereinafter referred to as stereo matching) is a popular research topic for decades to find a match between two rectified images. In general, the stereo matching method consists of four steps: cost calculation, cost aggregation, inconsistency calculation, and discrepancy resolution.

Cost aggregation plays an important role in local or global stereo matching. Most of the cost aggregation methods share a concept similar to joint image filtering using inductive images. Therefore, an inductive image is required. At this time, it is assumed that similar colors of the inductive image have similar disagreement values. However, if there is a highly textured area with a similar discrepancy value, this assumption often fails.

Reference: Korean Patent Publication No. 10-2015-0121179, Korean Patent Publication No. 10-2014-0030635

It is possible to provide a magnetic induction cost aggregation method and system for acquiring an accurate disparity map (Disparity-Map) by performing stereo matching based on a cost volume slice using a deep convolutional neural network.

An in-depth magnetic induction cost aggregation method for stereo matching, comprising: matching a stereo image based on a cost volume slice; And aggregating the magnetic induction matching cost by acquiring the disparity map by matching the stereo image.

Wherein matching the stereo image comprises: executing a deep convolutional neural network (DCNN) to match the stereo image based on the generated cost volume slice, wherein the deep convolutional neural network A deep convolutional neural network (DCNN) can be composed of a dynamic weight network and a descending filtering network.

The deep convolutional neural network (DCNN) uses a D convolution layer with a rectified linear unit (ReLU) layer along the first D-1 layer, and all but the first and last layers There are 64 kernels of size 3 x 3 x 64 for the layer, the first convolution layer consists of 64 kernels of 3 x 3 x 1, and the final convolution layer is a 3 x 3 x 64 size filter And the number of convolution layers can be set according to the size of the support area.

The step of matching the stereo image may include measuring the weight of the cost volume slice using the deeper convolution neural network and concurrently performing weight addition.

The step of matching the stereo image may include performing a weighted cost aggregation by selecting a cut absolute difference of color and gradient in the cost volume slice to match the cost calculation.

The step of aggregating the magnetic induction matching cost by acquiring the disparity map by matching the stereo image may include performing a consistency check on the left disparity map and the right disparity map to determine whether a pixel having a closed and an incorrect disparity exists For example.

An in-depth magnetic induction cost aggregation system for stereo matching, comprising: a matching unit for matching the stereo image based on a cost volume slice; And an aggregation unit for aggregating the magnetic induction matching cost by acquiring the disparity map by matching the stereo image.

The disparity map can be acquired more quickly and accurately by matching the stereo image based on the cost volume slice through the deep convoluted neural network without using the guidance image.

1 is a block diagram illustrating a configuration of a magnetic induction cost aggregation system according to an embodiment.
2 is a flow chart for explaining a method of self-induction cost aggregation in a magnetic induction cost aggregation system according to an embodiment.
3 is a diagram illustrating a general outline of a method of deep magnetic induction cost aggregation in a magnetic induction cost aggregation system according to an embodiment.
4 is a diagram illustrating an example of training data in a magnetic induction cost aggregation system according to an embodiment.
FIG. 5 shows an example of a comparison between data performed in the magnetic induction cost aggregation system according to an embodiment and conventional data.
FIG. 6 is a diagram for explaining a general outline of a deep learning network proposed in a self-induction cost aggregation system according to an embodiment.
FIG. 7 illustrates an example of an input image and a disparity map in the magnetic induction cost aggregation system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram for explaining a configuration of a magnetic induction cost aggregation system according to an embodiment, and FIG. 2 is a flowchart for explaining a magnetic induction cost aggregation method in a magnetic induction cost aggregation system according to an embodiment.

The magnetic induction cost aggregation system 100 may include a matching unit 110 and an aggregation unit 120. The components of the magnetic induction cost aggregation system 100 may control the magnetic induction cost aggregation system 100 to perform the steps 210 to 220 that are included in the magnetic induction cost aggregation method of FIG.

In step 210, the matching unit 110 may match the stereo image based on the cost volume slice. The matching unit 110 may execute a deep convolutional neural network (DCNN) to match the stereo image based on the cost volume slice. The matching unit 110 can measure the weight of the cost volume slice and perform the weight addition at the same time using the deep convolutional neural network. The generating unit 120 may perform the weighted cost aggregation by selecting the absolute difference between the color and the gradient in the cost volume slice to match the cost calculation.

In step 220, the tallying unit 120 can compute the self-induced matching cost by acquiring the disparity map by matching the stereo image. The tallying unit 120 may perform a consistency check on the left disparity map and the right disparity map to detect a pixel having an obstruction and an incorrect disparity.

3 is a diagram illustrating a general outline of a method of deep magnetic induction cost aggregation in a magnetic induction cost aggregation system according to an embodiment.

The magnetic induction cost aggregation system can select the absolute difference in color and gradient as the matching cost model. Weighted cost aggregation can be performed by matching cost calculations. Generally, an inductive image is needed to measure the weight for each pixel of the window. In an embodiment, a cost volume slice can be used instead of using an inductive color image. Accordingly, it is possible to measure the weights by applying the in-depth learning technique and to perform the weight addition at the same time.

Finally, a Winner-Take-All (WTA) strategy can be implemented to obtain the disparity value. After calculating the left disparity map and the right disparity map, left and right coherence checks can be used to detect occlusion and pixels with incorrect mismatches. To fill the holes, the lowest disparity value of the closest pixel with the correct disparity in the same scan line can be assigned. The disparity can then be modified using an adaptive median filter to preserve the depth boundary.

Referring to FIG. 3, an outline of the network structure is shown. The structure of the deep convoluted neural network can use D convolutional layers where the rectified linear unit (ReLU) layer follows the first D-1 layer.

You can include 64 kernels of size 3 x 3 x 64 for all layers except the first and last. This means that a small filter is performed on the 64 channels of each layer. The first convolution layer consists of 64 kernels, 3 x 3 x 1, and the last convolution layer can consist of 3 x 3 x 64 size filters.

The number of convolution layers can be set according to the size of the support area desired to be handled. For example, for 19 x 19 support areas, D = 9 can be set so that the center pixel in the support area can affect all other pixels.

4 is a diagram illustrating an example of training data in a magnetic induction cost aggregation system according to an embodiment.

Referring to FIG. 4, an example of a training data set in two areas is shown. The first row and the second row are the depth discontinuity patches around, and the third row shows the uniform depth patches.

Cost Because it is difficult to actually measure the volume slice, it is difficult to utilize a deep learning network for the cost aggregation step. So that the derived filtered cost volume can be used. At this time, only the filtered cost volume causing the correct disparity can be selected.

Since the deep learning network proposed in the embodiment is end-to-end learning, a series of patches can be generated as a data set instead of a series of images. There are two sets of data sets, a depth discontinuous patch and a uniform depth patch. The ratio of these two sets is 50:50, so the trained parameters can work normally in both areas.

FIG. 5 shows an example of a comparison between data performed in the magnetic induction cost aggregation system according to an embodiment and conventional data.

To compare the performance of the proposed deep magnetic induction cost aggregation method (DEEP) proposed in the embodiment, it is possible to compare methods such as conventional GF and conventional NL. You can also compare the results with the simple average filter (BOX).

FIG. 5 shows a qualitative comparison of a books data set, showing a disparity map of enlarged / reduced book data. (a) is the left image input, (b) is the actual data, (c) is BOX, (d) is GF, (e) is NL and (f) is the proposed method (DEEO). It can be seen from this that the proposed method performs well in a highly textured area with similar disparities. The red box shown in FIG. 5 indicates a well-textured area and a zoom version, so that it can be understood well.

The weights based on the inductive image may not be the same as the weights based on the observed disparity in the highly textured region.

While the existing methods do not obtain accurate and smooth discrepancies in the domain, the proposed deep magnetic induction method can obtain better results.

Table 1 shows the percentage of defective pixels in each data set.

Table 1:

The defective pixel ratio can be calculated using the non-occluded area. The proposed method is shown to be able to obtain comparable results using other conventional cost aggregation methods. The method according to the embodiment is free from the hand cap of the inductive image.

The present invention proposes a method of aggregating the cost of magnetic induction to obtain accurate stereo matching results. First, we randomly generate training patches in the training data set, and execute deep convolutional neural networks to learn the weight parameters. At this time, instead of using the inductive image, the cost volume slice can be guided by itself to perform the cost aggregation.

FIG. 6 is a diagram for explaining a general outline of a deep learning network proposed in a self-induction cost aggregation system according to an embodiment.

A magnetic induction cost aggregation system may propose a deep cost aggregation network (e. G., Deep convergent neural network) for performing magnetic induction cost aggregation. The deep cost aggregation network proposed in the embodiment can be composed of two sub-networks including a dynamic weighting network and a down-filtering network. The dynamic weighting network generates a series of dynamic weights for each input pixel, and these weights are useful as guidelines for filtering networks. The down-filtering network may perform an edge-aware filtering operation on the input cost volume slice. This results in the use of an edge-aware filtered cost volume slice.

Referring to FIG. 6, an outline of a deep magnetic induction cost aggregation network is shown, where W and H mean the input width and height. Also, d and r mean the diameter and radius of the support area. The loss function may consist of a feature reconstruction loss and a mean square loss function

FIG. 7 illustrates an example of an input image and a disparity map in the magnetic induction cost aggregation system according to an embodiment. For example, the 2015 KITTI training data, the left image input on the left, and the disparity map of the proposed method on the right.

It is difficult to apply a deep learning network because there is no actually measured cost volume slice. Thus, the derived filtered cost volume can be used. At this time, the actually measured disparity map can be used as a guide.

In an embodiment, 80% of the patches around the depth discontinuity can randomly extract 1000 patches in each training image. At this time, data can be increased in consideration of rotation and contrast invariance.

Referring to FIG. 7, an evaluation can be performed to verify the performance using the Middlebury data set and the KITTI data set. Table 2 shows a comparison of average defective pixel ratios of 200 KITTI training images.

The same rating scale as the KITTI benchmark can be used. From Table 2, it can be seen that the proposed method (DEEP) achieves the smallest mean error.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (7)

A method of in-depth magnetic induction cost aggregation for stereo matching,
Matching the stereo image based on a cost volume slice; And
And calculating a self-induced matching cost by acquiring a disparity map by matching the stereo image
Lt; / RTI >
Wherein matching the stereo image comprises:
Performing a deep convolutional neural network (DCNN) to match the stereo image based on the cost volume slice
Lt; / RTI >
The deep convolutional neural network (DCNN) is composed of a dynamic weight network and a descending filtering network
Depth Self Induction Cost Aggregation Method. delete The method according to claim 1,
The deep convolutional neural network (DCNN)
The rectified linear unit (ReLU) layer uses D convolution layers along the first D-1 layer,
There are 64 kernels of size 3 x 3 x 64 for all layers except the first and last layer,
The first convolution layer consists of 64 x 3 x 3 x 1 kernels, the last convolution layer consists of 3 x 3 x 64 size filters,
The number of convolution layers is set according to the size of the support area
Depth Self Induction Cost Aggregation Method. The method according to claim 1,
Wherein matching the stereo image comprises:
Measuring a weight of the cost volume slice using the deep convolutional neural network and simultaneously performing a weight addition;
Wherein the method comprises the steps of: A method of in-depth magnetic induction cost aggregation for stereo matching,
Matching the stereo image based on a cost volume slice; And
And calculating a self-induced matching cost by acquiring a disparity map by matching the stereo image
Lt; / RTI >
Wherein matching the stereo image comprises:
Performing a weighted cost aggregation by selecting a cut absolute difference of color and gradient in the cost volume slice to match the cost calculation
Wherein the method comprises the steps of: A method of in-depth magnetic induction cost aggregation for stereo matching,
Matching the stereo image based on a cost volume slice; And
And calculating a self-induced matching cost by acquiring a disparity map by matching the stereo image
Lt; / RTI >
The step of aggregating the magnetic induction matching cost by acquiring the disparity map by matching the stereo image,
Performing a coherence check on the left disparity map and the right disparity map to detect a pixel having an obstruction and an incorrect disparity
Wherein the method comprises the steps of: 1. An in-depth magnetic induction cost aggregation system for stereo matching,
A matching unit for matching the stereo image based on the cost volume slice; And
And an aggregation unit for aggregating the magnetic induction matching cost by acquiring a disparity map by matching the stereo image,
Lt; / RTI >
The matching unit,
Performing a deep convolutional neural network (DCNN) to match the stereo image based on the cost volume slice
≪ / RTI >
The deep convolutional neural network (DCNN) is composed of a dynamic weight network and a descending filtering network
Magnetic induction cost aggregation system.

Images