KR102219561B1 - Unsupervised stereo matching apparatus and method using confidential correspondence consistency - Google Patents

Abstract

The present invention includes two convolutional neural networks (CNNs) that have the same structure and the same weight and are pre-trained by an unsupervised learning method, an encoder that extracts feature maps from an input stereo image, and between feature maps. A matching cost calculator that calculates a matching cost volume and a disparity that minimizes the matching cost volume among disparity candidates having a predetermined maximum disparity range is obtained for each pixel, and a disparity map is generated from the obtained disparity. Including a parity map acquisition unit, the two CNNs estimate positive samples based on correspondence point consistency according to epipolar constraints with respect to the disparity map obtained from the stereo image input at the time of training, and use the estimated positive samples as adjacent pixels. A stereo matching apparatus and method for learning by backpropagating an error between learning maps generated by propagation and disparity maps can be provided.

Description

Unsupervised learning stereo matching device and method based on correspondence point consistency {UNSUPERVISED STEREO MATCHING APPARATUS AND METHOD USING CONFIDENTIAL CORRESPONDENCE CONSISTENCY}

The present invention relates to a stereo matching apparatus and method, and to a stereo matching apparatus and method of an unsupervised learning method based on correspondence point consistency.

Stereo matching is a method for recognizing 3D geometrical composition from an image, and is used in various fields including stereo image reconstruction of computer vision systems, autonomous driving, advanced driver assistance systems (ADAS), robotics, etc. Has become.

Stereo matching is a technique for estimating 3D location information (depth information) from a stereo image having two different viewpoint images, and calculating the matching cost to measure dissimilarity between correspondence points in a stereo image This is the core process of the stereo matching technique.

However, the calculation of the matching cost is difficult due to the inherent matching ambiguity of the image itself, such as an occlusion area, a textureless area, or a change in lighting of a stereo image.

In order to overcome this difficulty in calculating the matching cost, various techniques have been proposed, and recently, a method of calculating the matching cost from an image using an artificial neural network learned by a deep learning technique has been proposed. In particular, among artificial neural networks, a convolution neural network, which exhibits excellent performance in the image processing field, is mainly used for calculating the matching cost.

When an artificial neural network is used to calculate the matching cost, the artificial neural network must be learned in advance to minimize the matching cost between corresponding points in a stereo image. However, in order for the artificial neural network to reliably calculate the matching cost, supervised learning, which requires a large amount of training images including learning labels, must be performed.

The learning label can be obtained by measuring a depth value that has been directly verified (ground truth) using a 3D sensor such as structured light, light detection and ranging, and a laser scanner. However, measurement using a 3D sensor not only requires high cost, but also has various errors depending on the conditions of use, so that manual error correction is additionally required.

Accordingly, there is a problem in that the number of training images provided with a learning label for learning is insufficient, and the artificial neural network is not normally trained, so that the matching cost cannot be normally estimated.

Korean Patent Registration No. 10-1354387 (registered on January 15, 2014)

An object of the present invention is to provide a stereo matching apparatus and method using an artificial neural network learned in an unsupervised learning method that does not require a learning label.

Another object of the present invention is to provide a stereo matching apparatus and method using a learned artificial neural network to minimize matching cost by using epipolar constraints and correspondence consistency between input stereo images.

Still another object of the present invention is to provide a stereo matching apparatus and method capable of performing reliable stereo matching in various environments such as lighting changes and outdoor environments.

The stereo matching apparatus according to an embodiment of the present invention for achieving the above object includes two convolutional neural networks (CNN) that have the same structure and the same weight and are pre-trained by an unsupervised learning method, and input An encoder for extracting feature maps from the stereo image that is being converted; A matching cost calculator configured to calculate a matching cost volume between the feature maps; And a disparity map obtaining unit that obtains a disparity for each pixel that minimizes a matching cost volume among disparity candidates having a predetermined maximum disparity range, and generates a disparity map from the obtained disparity. Including, the two CNNs estimate a positive sample based on the correspondence point consistency according to the epipolar constraint with respect to the disparity map obtained from the stereo image input at the time of training, and propagate the estimated positive sample to adjacent pixels. It is learned by backpropagating an error between the generated training maps and the disparity map.

The two CNNs estimate a sparse positive sample map by estimating a pixel whose distance between the corresponding points is less than a predetermined threshold sense based on the correspondence point consistency according to the epipolar constraint with respect to the disparity map, and obtain a sparse positive sample map. An interpolation map is generated by propagating the sparse positive samples from the sample map to adjacent pixels by an interpolation technique, and the training samples are estimated again according to the correspondence point consistency for the interpolation map, and the training map is generated from the estimated training samples. Can be.

The interpolation map may be generated by using a color of an input stereo image as a guide according to a color similarity constraint in the sparse positive sample map and propagating the sparse positive sample to adjacent pixels.

The two CNNs are learned by updating the weight by backpropagating the error between the disparity map input at the time of training and the training map, and iteratively learning by repeatedly generating a training label so that the error is within a predetermined reference error. Can be.

The stereo matching method according to another embodiment of the present invention for achieving the above object is input using two convolutional neural networks (CNNs) that have the same structure and the same weight and are pre-trained in an unsupervised learning method. Extracting feature maps from the stereo image; Calculating a matching cost volume between the feature maps; And obtaining a disparity for minimizing a matching cost volume among disparity candidates having a predetermined maximum disparity range for each pixel, and generating a disparity map from the obtained disparity. Including, the two convolutional neural networks are generated by estimating a positive sample based on correspondence point consistency according to an epipolar constraint with respect to the disparity map obtained from the input stereo image, and propagating the estimated positive sample to adjacent pixels. It is learned by backpropagating an error between the learning maps and the disparity map.

Accordingly, the stereo matching apparatus and method according to an embodiment of the present invention estimate a positive sample using epipolar constraints and correspondence point consistency between input stereo images, and obtain a learning sample map by propagating the estimated positive sample, It can be learned in an unsupervised learning method. In addition, it is possible to perform reliable stereo matching in various environments such as lighting changes or outdoor environments.

1 shows a schematic configuration of a stereo matching device according to an embodiment of the present invention.
2 shows a detailed configuration of the learning map generator of FIG. 1.
3 is a diagram for explaining an operation of each configuration of the stereo matching device of FIG. 1.
4 shows an example of a learning map generated according to an embodiment of the present invention.
5 shows a schematic configuration of a stereo matching method and a learning method thereof according to an embodiment of the present invention.
6 shows in detail the learning map generation and learning steps of FIG. 5.
7 shows the algorithm of the learning map generation and learning step of FIG. 6.
8 shows an example of a map generated in the process of generating a learning map according to an embodiment of the present invention.
9 is a diagram illustrating a comparison between a learning sample according to an embodiment of the present invention and a sample obtained by another unsupervised learning method.
10 to 12 are diagrams comparing training samples according to the structure of a CNN according to an embodiment of the present invention.
13 and 14 are views comparing stereo matching results according to the structure of a CNN according to an embodiment of the present invention.
15 and 16 show the results of simulating the robustness of lighting in the unsupervised learning method according to the present embodiment.
17 and 18 show simulation results of stereo matching performance in an outdoor driving environment.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the implementation of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless specifically stated to the contrary. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean units that process at least one function or operation, which is hardware, software, or hardware. And software.

FIG. 1 shows a schematic configuration of a stereo matching device according to an embodiment of the present invention, FIG. 2 shows a detailed configuration of a learning map generator of FIG. 1, and FIG. 3 shows an operation of each configuration of the stereo matching device of FIG. It is a drawing for explanation.

Referring to FIG. 1, a stereo matching device 100 according to the present embodiment includes a stereo image input unit 110, an encoder 120, a matching cost calculation unit 130, a disparity map acquisition unit 140, and a learning map. Includes part 150.

The stereo image input unit 110 acquires a stereo image to be subjected to stereo matching. Here, the stereo image is an image that can be acquired from a stereo camera, and may be composed of two images having different viewpoints. The stereo image may be obtained as a vertical image, left and right image, etc., depending on the structure of the stereo camera, but here, as an example, as shown in FIG. 3, a left image (I ^l ) and a right image (I ^r ) Is assumed to be obtained.

^{Accordingly, the encoder 120 obtains two feature maps (A l} , A ^r ) to minimize the matching cost by encoding the ^{left image (I l} ) and the right image (I ^r ) acquired from the stereo image input unit 110 do.

The encoder 120 includes two pre-trained convolution neural networks (CNNs), as shown in FIG. 3, and the two CNNs included in the encoder 120 have the same structure and have the same weight. (w) is applied and implemented as a siamese CNN that performs a feed forward process. The two CNNs of the encoder 120 are learned in advance according to a designated pattern recognition technique, so that the left feature map (A ^l ) and the right feature map (A ^{l) and the right feature map (A l) are input images from the left image (I l} ) and the right image (I ^{r ). r} ) is obtained.

Since the two CNNs included in the encoder 120 perform the feed forward process (Fw) by applying the same weight (w), each pixel (i) of the ^{two feature maps (A l} , A ^{r) is Equation 1} Is obtained as

The matching cost calculation unit 130 receives the feature maps A ^l and A ^r output from the encoder 120 and applies a matching cost volume C ^{l between feature maps A l} and A ^r. , C ^r ) is calculated as in Equation 2.

(Here, ∥·∥ ₁ represents the l ₁ -norm function.)

Equation (2) is similarly left characteristic map (A ^l) L matching cost volume (C ^l) eoteuna indicate the equation for calculating, left matching cost volume (C ^r) to the left feature map (A ^r) of the Can be calculated.

^{When the matching cost volume (C l} , C ^r ) is calculated by the matching cost calculation unit 130, the disparity map acquisition unit 140 is a _{maximum disparity range (d max} ) for all pixels (i). ) Of the disparity candidates (d={1, ..., d _max }), the disparity (d) whose matching cost volume (C ^l , C ^r ) is minimized is searched and obtained according to Equation 3, Create a disparity map.

In Equation 3 as an example, only the equation ^{for calculating the left disparity d l} is disclosed, but the right disparity d ^r can be similarly calculated.

According to Equation 3, the disparity map acquisition unit 140 acquires a disparity d having a minimum matching cost in a disparity search range in a winner-takes-all (WTA) method.

That is, the stereo matching apparatus 100 according to the present embodiment uses a Siamese CNN having the same structure and the same weight (w), so that I ^l (i _x , i _y ) ^{from the input stereo images (I l} , I ^r) = I ^r (i _x -d ^l (i), i _y ) is determined, and the left disparity (d ^l (i)) for each pixel (i) is estimated. .

At this time, the stereo matching device 100 estimates a suitable left disparity (d ^l (i)) from among disparity candidates (d = {1, ..., d _{max }) having a maximum disparity range (d max ).} To do this, calculate the matching cost (C ^l (i)) ^{between I l} (i _x , i _y ) and I ^r (i _x -d, i _{y ).} Here, the left disparity (d ^l (i)) may be determined to minimize the matching cost (C ^{l (i)) according to the WTA technique.} In addition, a left disparity map d ^l may be obtained from the obtained ^{left disparity d l (i).}

The right disparity map d ^r can also be obtained in a similar manner.

In the above, it has been described that the stereo matching device 100 ^{acquires both the left disparity map (d l} ) and the right disparity map (d ^r ). However, the left disparity map (d) is accurately used for a stereo image that does not contain noise. ^{If l} ) and the right disparity map d ^r are obtained, the left disparity map d ^l and the right disparity map d ^r have a mutually symmetric structure. Accordingly, the stereo matching apparatus 100 ^{may output at least one of the left disparity map d l} and the right disparity map d ^{r as a} disparity map d that is a stereo matching result.

Meanwhile, in order to calculate the matching cost using the CNN, as described above, the CNN must be learned in advance, and a large amount of training images including the training label are required. However, it is difficult to acquire a large amount of training images including the training label, and therefore, it is not easy to train the CNN.

Accordingly, in this embodiment, the learning map generator 150 does not request a training image including a training label when learning CNN by the encoder 120, and generates a training label using a stereo image that does not include the training label. ) Is additionally suggested.

The learning map generator 150 is a component that generates a learning label (or a learning image including a learning label) added to learn the Siamese CNN of the encoder 120. That is, as a configuration included in the learning device for training the stereo matching device 100 according to the present embodiment, if the Siamese CNN of the encoder 120 has already been learned, it may be excluded.

During learning, the stereo image input unit 110 may acquire a plurality of stereo images in order to learn the Siamese CNN of the encoder 120. Here, as an example, it is assumed that the stereo image input unit 110 acquires T (where T is a natural number) stereo images.

In addition, during learning, the initial value of the weight w for the Siamese CNN of the encoder 120 is randomly selected from a predetermined range (eg, 0 <w ≤ 1).

During training, since the Siamese CNN of the encoder 120 does not function normally, the matching cost volume (C) and the disparity map (d ^l , d ^{) obtained from the feature maps (A l} , A ^{r) output from the encoder 120 r} ) is also unreliable.

However, the learning map generation unit 150 of the present invention has an epipolar constraint on the disparity maps (d ^l , d ^{r ) obtained from the feature maps (A l} , A ^{r) output from the untrained encoder 120.} A learning map capable of learning the Siamese CNN of the encoder 120 is generated by estimating a reliable positive sample and propagating the estimated positive sample based on Correspondence Consistency according to epipolar constraint).

The learning map generation unit 150 repeatedly generates a learning map for the same input stereo image, thereby gradually increasing the reliability of the learning map. And by generating a learning map for each of a plurality of stereo images, the Siamese CNN can reliably extract the ^{feature maps (A l} , A ^{r) for minimizing the matching cost for various stereo images.}

Referring to FIG. 3, the learning map generation unit 150 may include a positive sample extraction unit 151, an interpolation map generation unit 153, and a learning map acquisition unit 155.

The positive sample extraction unit 151 receives the left disparity map (d ^l ) and the right disparity map (d ^r ) obtained from the disparity map acquisition unit 140, and receives the left disparity map (d ^l ) and the right For each disparity (d ^l (i), d ^r (i)) of the disparity map (d ^r ), positive samples are searched and extracted based on the correspondence point consistency according to the epipolar constraint.

In stereo matching, pixels of the left image cross an epipolar line and have at most one matching pixel in the right image, and vice versa. That is, in order to match the pixels from the left image to the right image, the corresponding point (pixel) of the right image must also match the left image.

When the correspondence point consistency according to the epipolar constraint is used, a reliable positive sample can be extracted from the ^{left disparity map (d l} ) and the right disparity map (d ^{r) as shown in Equation 4.}

Where t is a predetermined threshold.

)로 추정된다.That is, according to Equation 4, a ^{pixel whose distance between corresponding points in the left disparity map (d l} ) and the right disparity map (d ^r ) is less than the threshold value (t) is a sparse positive sample (

).

) 이외의 나머지 픽셀은 일관성이 없는 불량 픽셀로 판단하여 제거한다.And the positive sample extraction unit 151 is the estimated rare positive sample (

Pixels other than) are determined as inconsistent defective pixels and removed.

)은 모든 영상 도메인에 분포될 수 있으나, 객체의 경계와 질감없는(textureless) 영역의 모호함으로 인해 오류 샘플이 포함될 수 있으며, 이로 인해 학습 성능 저하를 유발할 수 있다.At this time, the estimated rare positive sample (

) May be distributed in all image domains, but an error sample may be included due to the ambiguity of the boundary of the object and the textureless region, which may cause a decrease in learning performance.

)로 추정할 수 있다.Accordingly, the positive sample extraction unit 151 additionally adds a color similarity constraint that the positive samples estimated from each ^{of the left disparity map (d l} ) and the right disparity map (d ^{r) have similar color values due to the correspondence point consistency.} Can be applied. In this case, only the positive sample whose color difference is less than the predetermined reference value in the estimated positive sample is a rare positive sample (

) Can be estimated.

)을 주변 픽셀로 전파하여 보간 맵(p)을 생성한다. 보간 맵 생성부(153)는 입력 컬러 영상을 가이드로 이용하여 희소 양성 샘플(

)을 인접 픽셀로 전파하는 보간 기법을 적용한다.And the interpolation map generator 153 is a sparse positive sample (

) Is propagated to surrounding pixels to generate an interpolation map (p). The interpolation map generation unit 153 uses the input color image as a guide to generate a rare positive sample (

) Is applied to the adjacent pixels.

The interpolation map generator 153 obtains an interpolated pixel pi by performing interpolation so that the total energy function J(p) according to Equation 5 is minimized in the repeated learning process.

) 사이의 밸런스를 제어하기 위한 상수이며, h_i는 유효 픽셀인 경우 1이고 아니면 0을 나타내는 색인 함수이며, N₄(i)는 보간 픽셀(p_i)에 대한 인접 픽셀의 집합을 나타낸다. 그리고 범위 파라미터(σ_c)와 함께

로 정의되는 공간 변이 가중치 함수(w_i,j(I))를 이용하여 평활도 제약이 적응적으로 강제된다.Where λ is the sparse positive sample estimated from the two disparity maps (d ^l , d ^r)

) Is a constant for controlling the balance between, h _i is an index function representing 1 in the case of an effective pixel and 0 otherwise, and N ₄ (i) represents a set of adjacent pixels for the _{interpolation pixel p i.} And with the range parameter (σ _c )

The smoothness constraint is adaptively enforced using the spatial variation weight function (w _{i,j (I)) defined as.}

In addition, an interpolation map p, which is an interpolated disparity map obtained as an _{interpolation pixel p i, may be calculated by Equation 6.}

와

은 픽셀 수 S를 갖는 보간 맵(p)과 희소 양성 샘플 맵(

)의 S X 1 열 벡터를 나타내고, h는 희소 학습 샘플의 인덱스 벡터를 나타내며, m(m ∈ {0, ..., S-1})은 각 픽셀(i)에 대응하는 스칼라 인덱스(scalar index)를 나타낸다. 그리고 I는 항등 행렬, L은 수학식 7에 의해 정의되는 공간 변이 라플라시안 행렬을 나타낸다.here

Wow

Is an interpolation map (p) with the number of pixels S and a sparse positive sample map (

) Of SX 1 column vector, h is the index vector of the sparse learning sample, m(m ∈ {0, ..., S-1}) is the scalar index corresponding to each pixel (i) ). In addition, I denotes an identity matrix, and L denotes a spatially displaced Laplacian matrix defined by Equation 7.

Here, m and n denote a scalar index corresponding to a pixel, and N ₄ (m) denotes a set of pixels adjacent to pixel m.

)로서 획득한다. 이때 양성 샘플 추출부(151)는 학습 샘플(

)의 공간적 위치(Ω)를 함께 획득하여 학습 맵 획득부(155)로 전달한다.The interpolation map generation unit 153 transfers the obtained interpolation map (p) to the positive sample extraction unit 151, and the positive sample extraction unit 151 is an interpolation map (p) transmitted from the interpolation map generation unit 153 For the training sample (

). At this time, the positive sample extraction unit 151 is a learning sample (

The spatial location (Ω) of) is acquired together and transmitted to the learning map acquisition unit 155.

)과 위치(Ω)에 따라 학습 맵(

)을 획득한다.The learning map acquisition unit 155 includes a learning sample transmitted from the positive sample extraction unit 151 (

) And the learning map (

).

) 사이의 오차를 역전파함으로써, 샴 CNN의 가중치(w)가 학습 레이블(또는 학습 레이블이 포함된 학습 영상)을 기반으로 추정되는 오차에 따라 업데이트 되도록 한다. 즉 샴 CNN을 학습시킨다. 이러한 학습은 오차가 기지정된 기준 오차 이내로 감소될 때까지 반복적으로 수행될 수 있으며, 다수의 입력 스테레오 영상에 대해 각각 수행된다.In addition, the learning map acquisition unit 155 includes a disparity map (d) and a learning map obtained by the disparity map acquisition unit 140 from ^{the feature maps (A l} , A ^r ) acquired from the Siamese CNN of the current encoder 120. (

), the weight (w) of the Siamese CNN is updated according to an error estimated based on the training label (or the training image including the training label). That is, it trains Siamese CNN. Such learning may be repeatedly performed until the error decreases within a predetermined reference error, and is performed for each of a plurality of input stereo images.

)을 추출하고, 이후 보간 맵 생성부(153)가 보간 맵(p)을 생성하여 다시 양성 샘플 추출부(151)로 전달한다. 이에 양성 샘플 추출부(151)는 보간 맵(p)으로부터 학습 샘플(

)과 위치(Ω)를 추출하여 학습 맵 획득부(155)로 전달하고, 학습 맵 획득부(155)는 학습 샘플(

)과 위치(Ω)를 이용하여 학습 맵(

)을 획득한다.Referring to the detailed operation of the learning map generation unit 150 in FIG. 3, first, the positive sample extraction unit 151 is used for a sparse positive sample (

) Is extracted, and then, the interpolation map generation unit 153 generates an interpolation map p and transmits it to the positive sample extraction unit 151 again. Accordingly, the positive sample extracting unit 151 uses the learning sample (

) And the location (Ω) are extracted and transferred to the learning map acquisition unit 155, and the learning map acquisition unit 155 is a learning sample (

) And location (Ω) to learn map (

).

As described above, when the learning map generator 150 extracts a positive sample from the input stereo image using correspondence point consistency according to the epipolar constraint, and propagates the extracted positive sample to obtain a learning label, the learning label is There is an advantage that it is not necessary to acquire a separate training image included.

In addition, it is possible to learn to perform reliably stereo matching even for stereo images acquired in environments where lighting changes are strong, such as outdoors, and to show high stereo matching performance despite obscuration or ambiguity of texture-free areas. have.

,

,

)은 각각 좌 디스패리티 맵, 좌 희소 양성 샘플 맵 및 좌 학습 맵을 나타낸다. 도3 도시된 바와 같이, 초기 디스패리티 맵(

)에 비해 희소 양성 샘플 맵(

)에서 더욱 정확하게 객체가 표현됨을 알 수 있다. 그리고 학습 맵(

)은 희소 양성 샘플 맵(

)에서 누락된 픽셀이 보완됨으로써 영상 품질이 개선되었음을 알 수 있다.The image on the right in Fig. 3 (

,

,

) Denotes a left disparity map, a left sparse positive sample map, and a left learning map, respectively. As shown in Figure 3, the initial disparity map (

) Compared to the sparse positive sample map (

), you can see that the object is represented more accurately. And the learning map (

) Is the sparse positive sample map (

), it can be seen that the image quality is improved by supplementing the missing pixels.

), 좌 희소 양성 샘플 맵(

) 및 좌 학습 맵(

)을 도시하였으나, 학습 맵 생성부(150)에서는 우 디스패리티 맵(

), 우 희소 양성 샘플 맵(

) 및 우 학습 맵(

) 또한 유사하게 획득된다.Here, as a simple example, the left disparity map (

), left sparse positive sample map (

) And the left learning map (

) Is shown, but in the learning map generator 150, a right disparity map (

), right rare positive sample map (

) And right learning map (

) Is also obtained similarly.

4 shows an example of a learning map generated according to an embodiment of the present invention.

In Fig. 4, (a) shows the left image of the input stereo image, (b) shows the learning map generated according to this embodiment, and (c) shows the verification data (ground trouth) provided in the KITTI benchmark. Represents the generated disparity map.

As shown in FIG. 4, it can be seen that the learning map obtained from a stereo image without verification data according to the present embodiment provides a very good level of disparity map for learning compared to the disparity map including verification data. have.

Meanwhile, in FIG. 3, a structure of a Siamese CNN that generates ^{feature maps A l} and A ^{r by} receiving a corresponding image from a left image and a right image of a stereo image may be variously designed.

In this embodiment, as an example, we propose two structures, a simple CNN structure and a precise CNN structure.

First, the simple CNN structure is shown in Table 1. The simple CNN includes four convolution layers (conv1 to conv4) composed of a convolution kernel with a size of 3 X 3 for fast operation. In addition, ReLU (Rectifier Linear Unit) is applied to the three convolution layers (conv1 to conv3) as an activation function. ReLU allows only the positive part of the encoding result of each convolution layer (conv1 to conv3) to be transferred to the next layer.

_{In addition, an L2 normalization layer (l 2} -norm) is added instead of ReLU to the last convolution layer (conv4) to maintain the encoding result of negative numbers.

Correlation in Table 1 is the configuration of the matching cost calculation unit 130 of FIG. 1, and when the size of the stereo image input to obtain the matching cost volume C is hxw, the disparity candidate For (d={1, ..., d _max }), it requires calculation of _{hxwxd max times.}

The simple CNN shown in Table 1 can also exhibit a certain level of performance, but it is a single scale structure that does not adjust the size of the feature map output from each convolution layer (conv1 ~ conv4). There is a limit to the receptive field.

In the precise CNN structure shown in Table 2, a U-net structure including a sub-sampling layer and skip connection is used. This increases multi-scale information through the feed forward process.

In the precise CNN structure, a converging path and an extended path are included, and in the converging path, two 3 x 3 convolution kernels are successively applied, as in a general CNN, and batch normalization and After performing ReLU, max pooling may be performed for downsampling. At this time, the stride may be set to 2. In each downsampling step, the number of channels is doubled.

On the other hand, in the extension path, the upsampling and up-convolution of the feature map and the number of channels are halved for each step, and two 3 x 3 convolution kernels, batch normalization and ReLU are applied. _{In addition, the L2 normalization layer (l 2} -norm) is added instead of the batch normalization and ReLU in the last layer to maintain a negative encoding result.

In Table 2, the correlation is the configuration of the matching cost calculation unit 130 of FIG. 1, and calculates the matching cost volume C.

In the above, as an example of the structure of the Siamese CNN, two structures have been proposed, a simple CNN structure and a precise CNN structure, but the present invention is not limited thereto.

5 shows a schematic configuration of a stereo matching method and a learning method thereof according to an embodiment of the present invention, FIG. 6 shows in detail the learning map generation and learning steps of FIG. 5, and FIG. 7 is a learning map of FIG. Represents the algorithm of the generation and learning phase.

Referring to FIGS. 5 to 7, a stereo matching method and a learning method thereof according to the present embodiment will be described. First, in order to learn a stereo matching device, a plurality of learning stereo images are input to the stereo image input unit 110 (S10). ). Here, the plurality of learning stereo images simply mean stereo images used for learning, and are general stereo images that do not include a learning label. That is, in this embodiment, a separately produced learning stereo image including a learning label is not required for learning of the stereo matching device.

In addition, after acquiring the training disparity map (d) for each of the plurality of training stereo images in the same manner as the stereo matching method, a training map is generated and learned from the obtained disparity map (d) (S20).

Referring to FIGS. 6 and 7, when the learning map generation and training steps are described in detail, two Siamese CNNs having the same structure and the same weight (w) in which training is not completed in the encoder 120 are obtained from the training stereo image. Learning feature maps (A ^l , A ^r ) are extracted (S21). In addition, the matching cost calculation unit 130 calculates the learning matching cost volumes C ^l and C ^{r from the feature maps A l} and A ^r (S22).

The disparity map acquisition unit 140 is a matching cost volume among disparity candidates (d={1, ..., d _max }) ^{for all pixels i of the obtained matching cost volumes C l} , C ^r A learning disparity map (d) is generated by obtaining a disparity (d) in which (C ^l , C ^{r) is minimized according to Equation 3 (S23).}

From the step of extracting the learning feature map (S21) to the step of generating the disparity map (S23), it is the same as the method in which the stereo matching apparatus performs stereo matching. However, as described above, during learning, the Siamese CNN of the encoder 120 is different from the stereo matching operation in which the learning is not completed.

)을 추정한다(S24). 이때 양성 샘플 추출부(151)는 추정된 희소 양성 샘플(

) 이외의 나머지 픽셀은 제거한다.When the learning disparity map (d) is obtained, the positive sample extraction unit 151 of the learning map generation unit 150 performs a sparse positive sample on the basis of the correspondence point consistency according to the epipolar constraint with respect to the learning disparity map (d).

) Is estimated (S24). At this time, the positive sample extraction unit 151 is the estimated rare positive sample (

Pixels other than) are removed.

)을 인접 픽셀로 전파하는 보간 기법에 따라 보간 픽셀(pi)을 획득함으로써, 보간 맵(p)을 생성한다(S25).On the other hand, the interpolation map generator 153 uses the input color image as a guide to generate a rare positive sample (

An interpolation map (p) is generated by acquiring an interpolation pixel (pi) according to an interpolation technique that propagates) to adjacent pixels (S25).

)의 공간적 위치(Ω)를 추정한다(S26). 그리고 학습 맵 획득부(155)는 양성 샘플 추출부(151)에서 전달되는 학습 샘플(

)과 위치(Ω)에 따라 학습 맵(

)을 획득한다(S27).The positive sample extraction unit 151 is again based on the correspondence point consistency with respect to the interpolation map (p), and the learning sample (

The spatial position (Ω) of) is estimated (S26). In addition, the learning map acquisition unit 155 includes a learning sample transmitted from the positive sample extraction unit 151 (

) And the learning map (

) Is obtained (S27).

)이 획득되면, 디스패리티 맵 획득부(140)에서 획득된 디스패리티 맵(d)과 획득된 학습 맵(

) 사이의 오차를 역전파함으로써, 샴 CNN의 가중치(w)가 업데이트 되도록 한다(S28). 즉 샴 CNN을 학습시킨다.Learning map(

) Is obtained, the disparity map (d) obtained by the disparity map obtaining unit 140 and the obtained learning map (

) By backpropagating the error between, so that the weight (w) of the Siamese CNN is updated (S28). That is, it trains Siamese CNN.

The above-described learning may be repeatedly performed until the error decreases within a predetermined reference error, and is repeatedly performed for each of a plurality of input stereo images.

Referring back to FIG. 5, after the learning of the Siamese CNN of the encoder 120 is completed, a stereo image to be performed stereo matching is input to the stereo image input unit 110.

In addition, the two Siamese CNNs, which have been trained in the encoder 120 ^{, extract feature maps A l} and A ^r (S40). The matching cost calculation unit 130 calculates the matching cost volumes C ^l and C ^{r from the extracted feature maps A l} and A ^r (S50).

The disparity map acquisition unit 140 obtains the ^{disparity d in which the matching cost volumes C l} and C ^r are minimized according to Equation 3 and generates a disparity map d, thereby performing stereo matching. (S60).

Meanwhile, the learning map generator 150 according to an embodiment of the present invention removes inconsistent pixels using correspondence point consistency and includes an error backpropagation process, so that loss may occur. This allows for a softmax loss of each pixel for all possible disparity candidates.

In this embodiment, entropy loss is calculated for each pixel and disparity candidate, and the entropy loss is defined by Equation 8.

Where s is the value defined for all disparities as _{s x} = {i _x -1, ..., i _x -d _{max }, P T} (s;i) is 1 for the sampled training set, Otherwise, it is a label defined as 0, and P(s;i) is a soft max probability defined as in Equation 9.

Here, v is a value defined for all disparities _{as v x} = {i _x -1, ..., i _x -d _{max }, like s.} In Equation 9, a similarity score is converted into a matching cost by applying a negative sign.

Even if the learned CNN is used, the original matching cost volume may not be sufficient to generate an accurate disparity map. In particular, errors may occur in untextured areas and occluded areas. Accordingly, like the conventional stereo matching technique, a post-processing task may be further added.

Post-processing may include global matching and subsequent consistency checks, sub-pixel highlighting, and median and bidirectional filtering.

Semi-Global Matching reduces matching cost by applying a smoothness constraint condition to the disparity map.

The energy function E(d) of the disparity map d may be defined as in Equation 10.

는 매칭 비용이 높은 디스패리티에 패널티를 부가하고,

및

는 각각 불연속적인 디스패리티와 P₁ < P₂의 조건에 대해 패널티를 부가한다.In Equation 10

Imposes a penalty on disparity with high matching cost,

And

Is penalized for discontinuous disparity and P ₁ <P _{2, respectively.}

In the semi-global matching technique, an approximate value is obtained by performing one-dimensional cost update in a dynamic programming style according to various directions, and generally, optimization is performed according to two horizontal directions and two vertical directions. Accordingly, in order to minimize the energy function (E(d)) of Equation 10 in the direction r, the matching cost (Lr(i,d(i))) along the direction r is defined as in Equation 11.

After averaging costs for four directions using Equation 11, initial disparity maps for left and right directions may be calculated.

Thereafter, the initial disparity map may be improved through post-processing such as interpolation, sub-pixel enhancement, and smoothing. In the interpolation, a left-right consistency check is performed to eliminate a collision between a left disparity map and a right disparity map. In sub-pixel enhancement, a quadratic function is applied to adjacent pixels to obtain a disparity map in which the sub-pixels are emphasized.

Thereafter, in order to smooth the disparity map without blurring the edges, a 5 x 5 median filter and both filters may be sequentially applied.

Hereinafter, the results of simulation of the performance of the stereo matching apparatus and method according to the present embodiment are shown.

A simulation environment was constructed using VLFeat MatConNet Toolbox, and the stereo image for learning was a patch cut into a random 256 X 512 size from known KITTI, Middlebury, HCI, and Yonsei benchmark stereo images, and the maximum disparity range (d _max ) was set to 228, 116, 140, 80.

In KITTI 2012 and KITTI 2015, 194 and 200 pairs of stereo images, respectively, and 27 pairs of Middlebury stereo images between 2005 and 2006 were studied in an unsupervised method. In particular, in order to evaluate the reliability of lighting changes, Middlebury Stereo images with three different conditions were used in. In addition, to evaluate various outdoor driving conditions, HCI benchmarks including 11 challenge scenarios and 53 Yonsei benchmark stereo images with different time and season conditions were used.

In addition, the threshold value (t) is set to 1, the balance constant (λ) is set to 20 ² , and the range parameter (σ _c ) is set to 0.015. In addition, for learning optimization, the initial learning rate, momentum, and weight decay were set to 0.001, 0.9, and 0.005, respectively.

)을 이용한 비지도 학습 방법을 다른 비지도 학습 방법과 비교하여 시뮬레이션 한 결과로 MIDDLEBURY, KITTI 2012, 및 KITTI 2015 학습 데이터에 대한 비지도 학습 방법에서의 오차율의 정량적 비교 결과를 나타낸다.Table 3 shows the learning map generated according to this embodiment (

), compared with other unsupervised learning methods and simulated, and shows the result of quantitative comparison of the error rates in the unsupervised learning methods for MIDDLEBURY, KITTI 2012, and KITTI 2015 learning data.

In Table 3, the case of performing correspondence point consistency and propagation of positive samples in this example was compared and simulated together with the case of not performing (w/o PP), and a simple CNN was used. In the final matching cost volume, image sampling and soft-argmin of the spatial transformer network (STN) were applied, and a learning map was created using color consistency or color and disparity consistency (Col. Disp. Consistency). The cases were compared.

As shown in Table 3, when the unsupervised learning method according to the present embodiment is used, compared to other unsupervised learning methods, it shows better performance even when the correspondence point consistency and positive sample propagation are not performed. It can be seen that it shows more improved performance.

Table 4 shows the performance comparison results according to the structure of the CNN in the encoder 120 of this embodiment. Table 4 shows the results of comparing Census and MC-CNN fast, Content-CNN, and MC-CNN-WS with simple CNN and precision CNN in Tables 1 and 2, and as in Table 3, MIDDLEBURY, KITTI 2012, and KITTI Shows the result of a quantitative comparison of the error rate in the unsupervised learning method for 2015 learning data.

As shown in Table 4, when the CNN structure according to the present embodiment is used, it can be seen that even when a simple CNN is used, it shows better performance than other CNN structures, and when a precision CNN is used, it shows very excellent performance.

8 shows an example of a map generated in the process of generating a learning map according to an embodiment of the present invention.

)을 나타낸다.In FIG. 8, (a) shows the left image of the input stereo image input to the stereo image input unit 110, and (b) is the disparity map of the left image obtained by the disparity map acquisition unit 140 according to the WTA technique. And (c) is a sparse positive sample map obtained by using the correspondence point consistency in the positive sample extraction unit 151 of the learning map generator 150 (

).

)을 나타내며, (f)는 검증 자료 디스패리티 맵을 나타낸다.And (d) represents the interpolation map (p) interpolated by the interpolation map generator 153, (e) is obtained by using the correspondence point consistency again for the interpolation map (p) by the positive sample extraction unit 151 Learning map(

) And (f) represents the verification data disparity map.

9 is a diagram illustrating a comparison between a learning sample according to an embodiment of the present invention and a sample obtained by another unsupervised learning method.

)을 나타내며, (c)는 MC-CNN 스테레오 알고리즘과 신뢰도 임계값을 적용한 방법을 나타낸다.In Fig. 9, (a) shows the left image of the input stereo image, and (b) is a sparse positive sample map to which correspondence point consistency and total variation threshold are applied (

), and (c) shows the method of applying the MC-CNN stereo algorithm and the reliability threshold.

)을 나타내고, (i)는 검증 자료 디스패리티 맵을 나타낸다.(d) to (h) are learning maps obtained according to the number of repetitions when the initial random weight and the number of repetitions are set to 1, 3, 5, and 7 in this embodiment.

), and (i) represents the verification data disparity map.

)은 검증 자료 디스패리티 맵에 비교하여도 양호한 품질의 학습용 맵을 제공할 수 있음을 알 수 있다. 또한 학습 맵(

)을 획득하기 위한 과정을 반복함으로써, 더욱 우수한 학습 맵(

)을 획득할 수 있음을 알 수 있다.The learning map generated according to this embodiment in Figs. 8 and 9 (

) Can provide a map for learning of good quality even when compared to the validation data disparity map. Also learning map (

) By repeating the process to obtain a better learning map (

) Can be obtained.

10 to 12 are diagrams comparing training samples according to the structure of a CNN according to an embodiment of the present invention.

10 to 12 show results of obtaining a disparity map for the Middlebury, KITTI 2012 and KITTI 2015 data sets, respectively.

In each of Figs. 10 to 12, the left image represents the left image of the input stereo image, the middle image represents the disparity map obtained using the simple CNN structure of Table 1, and the right image represents the precise CNN structure of Table 2. Represents a disparity map obtained by using.

10 and 12, although it can be seen that a more accurate disparity map is obtained in a precise CNN structure than in a simple CNN structure, a disparity map that is good for use as a learning map in both the simple CNN structure and the precise CNN structure is used. You can see that you can get it.

13 and 14 are views comparing stereo matching results according to the structure of a CNN according to an embodiment of the present invention.

13 and 14 show disparity maps obtained as a result of stereo matching for KITTI 2012 and KITTI 2015 data sets, respectively. The left image shows the left image of the input stereo image, and the middle image shows the simple CNN structure of Table 1. The disparity map obtained using is shown, and the image on the right shows the disparity map obtained using the precise CNN structure of Table 2.

13 and 14, it can be seen that more accurate disparity maps are obtained in the precise CNN structure than in the simple CNN structure, but both the simple CNN structure and the precise CNN structure can obtain a disparity map as a good stereo matching result. Able to know.

Table 5 shows the result of comparison of stereo matching performance according to the structure of the CNN in the encoder 120 of the present embodiment. Table 4 shows the results of comparing the existing Guided Filter, Census + SGM, DLP, MC-CNN fast, MC-CNN acrt, Content-CNN, and MC-CNN-WS with the simple CNN and precision CNN in Tables 1 and 2. And shows the result of quantitative comparison of the error rate in the unsupervised learning method for KITTI 2012 and KITTI 2015 learning data.

15 and 16 show the results of simulating the robustness of lighting in the unsupervised learning method according to the present embodiment.

In order to analyze the effect of changes in lighting, the exposure index for all images was set to 1, and the lighting index was changed from 1 to 3. That is, it shows the simulation result for the case where the left and speculative illumination is 1/3.

In Fig. 15, the Middlebury data set was used, and the left two images represent the left and right images of the input stereo image, and then feature-based DASC (Dense Adaptive Self-Correlation) and color and disparity consistency in the right direction. (Col. Disp. Consistency), this embodiment and verification data disparity map are shown.

As can be seen from FIG. 15, it can be seen that the stereo matching device learned according to the unsupervised learning method according to the present embodiment exhibits very superior performance compared to other learning methods even when the lighting changes.

And, as shown in Fig. 16, the unsupervised learning method according to this embodiment has a matching cost similar to that of other learning methods even when using a simple CNN for a change in lighting, but when using a precise CNN, it shows a very low error rate. Can be seen.

17 and 18 show simulation results of stereo matching performance in an outdoor driving environment.

Fig. 17 shows the simulation result for the HCI data set, and Fig. 18 shows the simulation result for the Yonsei data set.

In Figs. 17 and 18, the left image shows the left image of the input stereo image, the middle image shows the disparity map obtained using the simple CNN structure in Table 1, and the right image shows the precision CNN structure in Table 2. Represents the obtained disparity map.

It can be seen from FIGS. 17 and 18 that the stereo matching apparatus and method learned by the unsupervised learning method according to the present embodiment exhibits excellent stereo matching performance even in an outdoor actual driving environment.

Table 6 shows the results of simulating the stereo matching speed of the stereo matching apparatus and method according to the present embodiment.

Table 6 shows the results of comparing the DLP technique with the case of using the simple CNN structure and the precise CNN structure encoder according to the present embodiment, and shows the simulation results for KITTI 2012, KITTI 2015 and MIDDLEBURY. In addition, simulation results for KITTI and MIDDLEBURY using patches of different resolutions are shown.

As shown in Table 6, the stereo matching method having a simple CNN structure according to the present embodiment operates slightly slower than DLP, but is superior to DLP in terms of stereo matching performance as described above. In addition, in the case of having a precise CNN structure, while the processing speed is slow, it exhibits very excellent stereo matching performance, and thus the required CNN structure can be selectively used in consideration of the balance between efficiency and accuracy.

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (Read Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, lighting data storage device, and the like.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those of ordinary skill in the art will appreciate that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Claims (10)

An encoder for extracting feature maps from an input stereo image, including two convolutional neural networks (CNNs) having the same structure and the same weight set by pre-performed learning;
A matching cost calculator configured to calculate a matching cost volume between the feature maps;
A disparity map obtaining unit that obtains a disparity for each pixel that minimizes a matching cost volume among disparity candidates having a predetermined maximum disparity range, and generates a disparity map from the obtained disparity; And
The learning disparity map output from the disparity map acquisition unit is applied to the learning stereo image, which is combined at the time of learning performed in advance to learn the encoder and input to the encoder at the time of learning. Generating a learning map that estimates a positive sample based on correspondence point consistency according to an epipolar constraint for a parity map and backpropagates the error between the learning maps generated by propagating the estimated positive sample to adjacent pixels and the learning disparity map Including wealth,
The learning map generation unit
The sparse positive sample is estimated as a sparse positive sample by estimating a pixel whose distance between the corresponding points is less than a predetermined threshold based on the consistency of correspondence points according to the epipolar constraint for the learning disparity map acquired by the disparity map acquisition unit during training. A positive sample extraction unit for obtaining a sample map;
An interpolation map generator configured to generate an interpolation map by propagating the sparse positive sample from the sparse positive sample map to adjacent pixels using an interpolation technique; And
A learning map for estimating a learning sample according to correspondence point consistency for the interpolation map again, generating the learning map from the estimated learning sample, obtaining an error between the generated learning map and the learning disparity map, and backpropagating it Including the acquisition unit,
The interpolation map generator
Using the color of the input stereo image as a guide, an interpolation pixel in which a predetermined energy function is minimized according to the color smoothness constraint between the color of the sparse positive sample and adjacent pixels in the sparse positive sample map is obtained, and the obtained interpolation A stereo matching device that generates the interpolation map using pixels.

delete delete delete The method of claim 1, wherein the learning map generator
A stereo matching device that performs iterative learning so that the error is within a predetermined reference error. A learning disparity map obtained by extracting learning feature maps from a learning stereo image input during learning, calculating a learning matching cost volume between the extracted learning feature maps, and searching for a disparity that minimizes the learning matching cost volume Estimating a positive sample based on the correspondence point consistency according to the epipolar constraint for, and performing learning by backpropagating an error between the learning maps generated by propagating the estimated positive sample to adjacent pixels and the learning disparity map ;
Extracting feature maps from an input stereo image using two convolutional neural networks (CNNs) having the same structure and the same weight preset according to an error backpropagated in the step of performing the learning;
Calculating a matching cost volume between the feature maps; And
Obtaining a disparity for minimizing a matching cost volume among disparity candidates having a predetermined maximum disparity range for each pixel, and generating a disparity map from the obtained disparity; Including,
The step of performing the learning
Obtaining a sparse positive sample map by estimating a pixel whose distance between corresponding points is less than a predetermined threshold value as a sparse positive sample based on correspondence point consistency according to an epipolar constraint with respect to the learning disparity map;
Generating an interpolation map by propagating the sparse positive sample from the sparse positive sample map to adjacent pixels by an interpolation technique; And
Estimating a learning sample according to correspondence point consistency with respect to the interpolation map again, generating the learning map from the estimated learning sample, obtaining an error between the generated learning map and the learning disparity map, and backpropagating it; Including,
Generating the interpolation map
Acquiring an interpolation pixel in which a predetermined energy function is minimized according to a color smoothness constraint between a color of a sparse positive sample and a color smoothness between adjacent pixels in the sparse positive sample map by using the input color of the stereo image as a guide; And
Generating the interpolation map using the obtained interpolation pixels; Stereo matching method comprising a.
delete delete delete The method of claim 6, wherein the learning step
Stereo matching method performed repeatedly so that the error is within a predetermined reference error.

Images