KR102483641B1 - Method and apparatus for processing binocular image - Google Patents

Abstract

A method and apparatus for processing a binocular parallax image are disclosed. A method for determining disparity of a binocular disparity image may include acquiring characteristics of a plurality of pixels included in the binocular disparity image based on a distribution of events of the binocular disparity image including a left eye image and a right eye image; calculating a cost matrix of pixels matched between the left-eye image and the right-eye image based on the characteristics of the pixels; and determining a parallax of each of the matched pixels based on the cost matrix.

Description

Binocular parallax image processing method and apparatus {METHOD AND APPARATUS FOR PROCESSING BINOCULAR IMAGE}

The following description relates to a processing technique of a binocular parallax image.

Regarding conventional binocular parallax image processing technology, a dynamic vision sensor (DVS) is a type of image sensor made of a complementary metal oxide semiconductor (CMOS). Images acquired by DVS generate events based on the magnitude of illumination change. The event portion is determined by comparing the magnitude of the lighting change to a threshold value. However, since the image acquired by DVS is sensitive to external influences, the event part has a lot of noise and the distribution and quantity of the event may not match.

A method for determining a disparity of a binocular disparity image according to an embodiment includes obtaining characteristics of a plurality of pixels included in the binocular disparity image based on a distribution of events of the binocular disparity image including a left eye image and a right eye image. The method may include calculating a cost matrix of pixels matched between the left-eye image and the right-eye image based on characteristics of the pixels, and determining a disparity of each of the matched pixels based on the cost matrix.

The obtaining of the feature may include obtaining a distribution of events by dividing a plurality of pixels included in the binocular parallax image into an event part and a non-event part, and each of the plurality of pixels included in the non-event part. Calculating a Euclidean distance from to the nearest pixel included in the event part, and setting each of the Euclidean distances as a feature of each of a plurality of pixels included in the non-event part. can

The calculating of the Euclidean distance may include obtaining parabolas of a plurality of functions representing Euclidean distances between a plurality of pixels included in the non-event part and a plurality of pixels included in the event part. The method may include obtaining intersection points between the parabolas and calculating a Euclidean distance to the nearest pixel based on an envelope below the intersection points.

The calculating of the cost matrix may include calculating a feature matching cost of pixels matched between the left eye image and the right eye image based on features of the pixels, polarity matching of the matched pixels based on polarities of the pixels The method may include calculating a cost and obtaining a cost matrix of the matched pixels based on the feature matching cost and the polarity matching cost.

In the determining of the disparity, the cost matrix may be filtered.

The method of determining disparity of the binocular disparity image may further include removing noise from the binocular disparity image.

The removing of the noise may include obtaining a plurality of feature vectors by applying orthogonal analysis to the binocular parallax image, calculating feature values from the plurality of feature vectors, and performing the binocular parallax image based on the feature values. A step of removing noise from the parallax image may be included.

The method of determining the disparity of the binocular disparity image may further include optimizing the obtained disparity.

The optimizing may include obtaining a correlation between parallaxes of the plurality of pixels and optimizing the obtained parallaxes based on the correlation.

The obtaining of the correlation may include obtaining a robustness value for each disparity by applying cross-validation to the disparity, and a correlation between all disparity based on the robustness value for each disparity. It may include the step of obtaining.

Optimizing based on the correlation may include obtaining a dense conditional random field based on the robustness value and the correlation, and optimizing the parallaxes using the dense conditional random field.

The optimizing using the dense conditional random field may include determining subpixel level disparity of the matched pixels based on the dense conditional random field, and photographing the subpixel level disparity and the binocular disparity image. The method may include obtaining depth values of the matched pixels based on the focal length of the camera.

An apparatus for generating a binocular parallax image, according to an embodiment, acquires characteristics of a plurality of pixels included in the binocular parallax image based on a distribution of events of the binocular parallax image including a left eye image and a right eye image. a matrix calculator for calculating a cost matrix of pixels matched between the left-eye image and the right-eye image based on characteristics of the pixels; and a disparity determiner for determining a disparity of each of the matched pixels based on the cost matrix. can include

The apparatus for determining disparity of the binocular disparity image may further include a noise removal unit that removes noise from the binocular disparity image.

The apparatus for determining the disparity of the binocular disparity image may further include an optimization unit that optimizes the obtained disparity.

1 is a diagram illustrating a situation in which binocular parallax images of an object are acquired using a plurality of cameras having different viewpoints according to an exemplary embodiment.
2 is a flowchart illustrating a sequence of a method of determining a disparity of a binocular disparity image according to an exemplary embodiment.
3 is a flowchart illustrating a procedure for acquiring features of pixels from a left-eye image and a right-eye image according to an exemplary embodiment.
4 is a diagram visualizing features extracted according to a method for determining disparity of a binocular disparity image according to an exemplary embodiment.
5 is a diagram visualizing disparity obtained according to a method for determining disparity of a binocular disparity image according to an embodiment.
6 is a diagram for comparing effects optimized according to an optimization method according to an embodiment.
7 is a diagram showing the overall configuration of an apparatus for determining disparity of a binocular disparity image according to an exemplary embodiment.
8 is a flowchart illustrating an operation performed by each component of a device for determining a disparity of a binocular disparity image according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be modified and implemented in various forms. Therefore, the embodiments are not limited to the specific disclosed form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

Hereinafter, an image acquired by DVS may be referred to as a DVS image. A binocular parallax image may be referred to as a stereo image.

1 is a diagram illustrating a situation in which binocular parallax images of an object are acquired using a plurality of cameras having different viewpoints according to an exemplary embodiment.

Camera 110 and camera 120 may be composed of DVS. The camera 110 and the camera 120 may be disposed at a phase difference by an angle θ to capture the object 100 .

For example, an x mark is marked in the center of the object 100 for explanation. The image 111 obtained by the camera 110 and the image 121 obtained by the camera 120 may represent the object 100 in different directions. In the image 111, the x mark is located in the center of the object 100, but in the image 121, the x mark is biased from the center of the object 100 to the left. This is because the camera 110 and the camera 120 photograph the object 100 with a phase difference of an angle θ.

In order to render a binocular parallax image, a parallax between two images of the same object 100 photographed by the cameras 110 and 120 must be determined. In the DVS image, pixels may be matched and parallax may be determined based on the magnitude of illumination change.

A portion in which pixels having a magnitude of illumination change equal to or greater than a threshold may be referred to as an event portion. In DVS video, events usually appear on the boundary of the object 100 or around the frame. Accordingly, an event in a DVS image may indicate information about the structure of the object 100 . In DVS video, the same object 100 usually shows a similar structure. Accordingly, an event corresponding to the left eye image and an event corresponding to the right eye image may have a similar structure.

DVS video requires special processing because there is a lot of noise in the event part and the distribution and number of events do not match. An apparatus for determining parallax of a binocular parallax image according to an embodiment can shorten calculation time because characteristics of pixels are quickly obtained by obtaining a correlation between pixels and events and obtaining a distribution of events using a simple algorithm.

2 is a flowchart illustrating a sequence of a method of determining a disparity of a binocular disparity image according to an exemplary embodiment.

According to an embodiment, in operation 210, the disparity determination device may remove noise from the binocular disparity image. Step 210 may be performed before step 220 . The binocular parallax image includes a left eye image and a right eye image.

In operation 211, the disparity determination device may obtain a plurality of feature vectors by applying orthogonal analysis to the binocular disparity image. In the DVS image, an event part is generated based on the size of the lighting change, and the event part is usually distributed around the boundary of the object 100 . Here, a portion with a sparse distribution of events may be determined as noise, and a portion with a dense distribution of events may be determined as a source for determining a disparity. The degree to which the distribution of events is dense can be referred to as strong correlation. The correlation of the distribution of events can be obtained using orthogonal analysis.

In step 213, the disparity determination device may calculate feature values from a plurality of feature vectors. The association relationship can be represented by a feature vector. A feature value of a feature vector corresponding to a distribution of events having a strong correlation is relatively large, and a feature value of a feature vector corresponding to a distribution of events having a small correlation is relatively small. A portion of an event corresponding to a feature vector having a small feature value may be determined as noise.

In operation 215, the disparity determination apparatus may remove noise from the binocular disparity image based on the feature values. The disparity determination apparatus may arrange feature values of feature vectors in order of size. The disparity determination apparatus may arrange the feature vectors in ascending order and determine a predetermined number of feature vectors in descending order of feature values as noise. Alternatively, the disparity determination apparatus may arrange the feature vectors in descending order and determine a predetermined number of feature vectors in descending order of feature values as noise. The parallax determination apparatus may obtain a DVS image from which noise is removed by combining the remaining feature vectors except for the feature vector determined to be noise.

Since the binocular parallax image may include a left eye image and a right eye image, a noise removal process may be performed on each image.

Orthogonal analysis is applied to the left eye image to obtain a feature vector, and feature values of the feature vector may be removed. After the feature vectors are sorted in ascending or descending order according to the size of feature values, a predetermined number of feature vectors in descending order of feature values may be determined as noise. The disparity determination apparatus may obtain a left eye image from which noise is removed by combining portions of the left eye image corresponding to the remaining feature vectors after removing the feature vector determined to be noise.

An orthogonal analysis is applied to the right eye image to obtain a feature vector, and feature values of the feature vector may be removed. After the feature vectors are sorted in ascending or descending order according to the size of feature values, a predetermined number of feature vectors in descending order of feature values may be determined as noise. The disparity determination apparatus may obtain a right eye image from which noise is removed by combining parts of the right eye image corresponding to the remaining feature vectors after removing the feature vector determined to be noise.

Hereinafter, a noise removal process is explained through Equation 1.

는 노이즈가 제거된 양안 시차 영상이며, e는 양안 시차 영상 내의 노이즈로 판단되는 부분이다. 여기서, 내림차순으로 특징 벡터가 정렬되는 경우, 파라미터 k는 특징값이 큰 순서대로 특징 벡터의 개수를 나타낸다. r은 특징 벡터의 전체 개수를 표시하고,

는 제i 번째 특징 벡터의 특징값을 표시하고, i는 특징벡터의 번호를 나타내며,

와

는 제i 번째의 직교하는 특징 벡터를 나타내고, H는 치환 오퍼레이션을 나타낸다. I에서 e를 제거함으로써

가 획득될 수 있다.Equation 1 represents an exemplary denoising formula. I is the input binocular parallax image,

is a binocular disparity image from which noise has been removed, and e is a portion determined to be noise in the binocular disparity image. Here, when the feature vectors are sorted in descending order, the parameter k represents the number of feature vectors in order of feature values. r denotes the total number of feature vectors,

denotes the feature value of the ith feature vector, i denotes the number of the feature vector,

Wow

denotes the ith orthogonal feature vector, and H denotes a permutation operation. By removing e from I

can be obtained.

According to an embodiment, in operation 220, the disparity determination apparatus acquires characteristics of a plurality of pixels included in the binocular disparity image based on the distribution of events of the binocular disparity image.

In operation 221, the disparity determination apparatus may obtain an event distribution by dividing a plurality of pixels included in the binocular disparity image into an event part and a non-event part. The characteristics of pixels included in the event part may be initialized. For example, characteristics of pixels included in the event part may be set to 0.

In operation 223, the disparity determination device may calculate a Euclidean distance from each of a plurality of pixels included in the non-event part to a nearest pixel included in the event part. The calculated Euclidean distance may represent characteristics of each of a plurality of pixels included in the non-event part.

In order to calculate the Euclidean distance to the nearest pixel, the parallax determination device uses a plurality of functions representing Euclidean distances between a plurality of pixels included in the non-event part and a plurality of pixels included in the event part. The parabolas of can be obtained. The parallax determination device may obtain points of intersection between the acquired parabolas. The parallax determination device may calculate the Euclidean distance to the nearest pixel based on the lower envelope of the intersection points.

In step 225, the parallax determination device may set each of the Euclidean distances as each characteristic of a plurality of pixels included in the non-event part.

In this way, the parallax determination device may replace the calculation of the feature for each pixel with the calculation of the Euclidean distance. Since the calculation of the Euclidean distance is a simple algorithm, the amount of calculation can be significantly reduced.

Since the binocular parallax image may include a left eye image and a right eye image, an algorithm for calculating a Euclidean distance may be applied to each image.

The parallax determination apparatus may obtain an event distribution by dividing a plurality of pixels included in the left eye image into an event part and a non-event part. The characteristics of pixels included in the event part may be initialized. For example, characteristics of pixels included in the event part may be set to 0. The disparity determination device may calculate a Euclidean distance from each of a plurality of pixels included in the non-event portion to a nearest pixel included in the event portion. The calculated Euclidean distance may represent characteristics of each of a plurality of pixels included in the non-event part of the left eye image.

The parallax determination apparatus may obtain an event distribution by dividing a plurality of pixels included in the right eye image into an event part and a non-event part. The characteristics of pixels included in the event part may be initialized. For example, characteristics of pixels included in the event part may be set to 0. The disparity determination device may calculate a Euclidean distance from each of a plurality of pixels included in the non-event portion to a nearest pixel included in the event portion. The calculated Euclidean distance may represent characteristics of each of a plurality of pixels included in the non-event part of the right eye image.

Hereinafter, the process of obtaining the Euclidean distance is explained through Equation 2.

는 픽셀 (x, y)과 가장 가까운 이벤트 (

)의 유클라디언 거리를 나타낸다. f(

)는 이벤트 (

)의 특징값을 표시한다. x는 픽셀 (x, y)의 횡좌표를 나타내고, y는 픽셀 (x, y)의 종좌표를 나타낸다. x'는 이벤트 (

)의 횡좌표를 나타내고, y'는 이벤트 (

)의 종좌표를 나타내며, n은 픽셀의 식별 번호이다. Equation 2 represents an exemplary Euclidean distance conversion formula. here,

is the event closest to pixel (x, y) (

) represents the Euclidean distance of f(

) is the event (

) is displayed. x represents the abscissa of the pixel (x, y), and y represents the ordinate of the pixel (x, y). x' is the event (

), and y 'is the event (

), and n is an identification number of a pixel.

), f(

))을 근으로 하는 포물선의 해를 구하는 문제로 볼 수 있다. 포물선의 해를 구하는 문제는 포물선 사이의 교점을 구하고 교점 이하의 영역을 구하는 문제로 이해될 수 있다. 이처럼, 유클라디언 거리를 구하는 문제는 포물선의 아래쪽 인벨로프 (lower envelope)의 해를 구하는 문제로 변환됨에 따라, 모든 픽셀에 대한 해를 구하는 문제는 최소의 포물선 교점의 집합을 구하는 문제로 변하게 된다.In order to directly obtain the Euclidean distance using Equation 2, it takes a long time to calculate because a quadratic equation must be obtained for each pixel. Essentially, Equation 2 is a plurality of ((

), f(

)) can be seen as a problem of finding the solution of a parabola. The problem of finding the solution of a parabola can be understood as a problem of finding the intersection between parabolas and finding the area below the intersection. In this way, as the problem of finding the Euclidean distance is transformed into the problem of finding the solution of the lower envelope of the parabola, the problem of finding the solution for all pixels becomes the problem of finding the minimum set of parabola intersections. do.

와

를 근으로 하는 포물선 사이의 교점은, 수학식 3으로 신속하게 확정될 수 있다. 여기서, i, j는 이벤트의 식별 번호이다. 이와 같이, 시차 결정 장치는 단순한 알고리즘을 이용함으로써 계산 시간이 단축되고 실시간으로 유클라디언 거리를 구할 수 있다.Determining the point of intersection between the Euclidean distance function parabolas can be relatively straightforward. For example,

Wow

The point of intersection between the parabolas rooted at can be quickly determined by Equation 3. Here, i and j are event identification numbers. In this way, the parallax determination device uses a simple algorithm, thereby reducing calculation time and obtaining the Euclidean distance in real time.

According to an embodiment, in operation 230, the disparity determination apparatus calculates a cost matrix of pixels matched between the left eye image and the right eye image based on the characteristics of the pixels. The cost matrix may include a feature matching cost and a polar matching cost.

In operation 231, the disparity determination apparatus may calculate a feature matching cost of pixels matched between the left eye image and the right eye image based on the feature of the pixels. For example, the feature matching cost may be derived using Equation 4.

는 매칭되는 픽셀(x, y)의 특징 매칭 비용을 나타낸다. W는 양안 시차 영상 내의 부분들의 집합을 나타내고,

는 W에 포함된 부분의 식별 번호를 나타내고, d는 현재 시차를 나타내며, N은 픽셀의 총 개수, n은 픽셀의 식별 번호를 나타낸다.here,

represents the feature matching cost of the matched pixel (x, y). W represents a set of parts in the binocular parallax image,

denotes an identification number of a part included in W, d denotes a current parallax, N denotes the total number of pixels, and n denotes an identification number of a pixel.

In operation 233, the parallax determination device may calculate a polarity matching cost of matched pixels based on the polarities of the pixels. For example, the polarity matching cost can be derived using Equation 5.

Here, CP(x, y, d) represents the polarity matching cost of the matched pixel (x, y). E(x, y) represents polarity within the coordinate system of pixel (x, y), and E(x+d, y) represents polarity within the coordinate system of pixel (x+d, y).

In operation 235, the parallax determination device may obtain a cost matrix of matched pixels based on the feature matching cost and the polarity matching cost. For example, the polarity matching cost can be derived using Equation 5.

Here, C(x, y, d) represents the cost matrix of the matching pixel (x, y). α is a linear weight.

In this way, the disparity determination device may convert a matching between events to a matching between dense pixels by converting a process of matching events into a process of matching pixels. Through this, the parallax determination device can be easily applied to specific application technologies such as 3D scene modeling and image rendering, which are subsequently possible.

According to an embodiment, in step 240, the disparity determination device determines a disparity of each matched pixel based on the cost matrix. For example, the disparity determining apparatus may determine disparity between matching pixels of the left eye image and the right eye image by using an algorithm such as a greedy strategy winner take all. For example, the disparity determination apparatus may determine the disparity between the left eye image and the right eye image using Equations 7 and 8.

는 좌안 영상의 비용행렬이고,

는 픽셀(x, y)의 시차이다.here,

is the cost matrix of the left eye image,

is the parallax of pixel (x, y).

는 우안 영상의 비용행렬이고,

는 픽셀(x, y)의 시차이다.here,

is the cost matrix of the right eye image,

is the parallax of pixel (x, y).

In the DVS video, the non-event portion usually has a lower illumination change than a preset threshold. Here, that the size of the illumination change of the pixels is similar means that viewpoints corresponding to the pixels are close to each other. Cost values of pixels included in adjacent non-event portions may be similar. It is necessary to make cost values of adjacent pixels of the non-event part similar by applying filtering in a smoothing filtering method to the cost matrix obtained in step 235.

In step 241, the disparity determination device may filter the cost matrix using a smoothing filtering method. The disparity determination apparatus may set a smoothing factor of the smoothing filtering method, and filter the cost matrix by the smoothing filtering method based on the smoothing factor. In operation 243, the disparity determination device may determine a disparity of each matched pixel based on the filtered cost matrix.

For example, the disparity determination device may filter the cost matrix using Equation 9 using a smoothing filtering method.

는 스무딩 필터링 방식으로 필터링된 비용 행렬을 나타낸다. p는 제p 번째 픽셀을 나타내고, d는 픽셀에 대응하는 시차를 표시하고; K는 근접한 부분에 속하는 픽셀의 개수를 나타낸다. q는 p에 근접한 부분에 속하는 픽셀을 나타내고,

는 스무딩 인자를 나타낸다.

는 수학식 10을 통해 계산될 수 있다.here,

Represents a cost matrix filtered by a smoothing filtering method. p denotes a p-th pixel, d denotes a parallax corresponding to the pixel; K represents the number of pixels belonging to the adjacent part. q represents a pixel belonging to a portion close to p,

represents a smoothing factor.

Can be calculated through Equation 10.

는 픽셀 p의 극성을 나타내고, v는 기 설정된 값이며 상수이다.here,

denotes the polarity of the pixel p, and v is a predetermined value and is a constant.

According to another embodiment, in step 250, the disparity determination device may optimize the obtained disparity. The disparity determination device may optimize the disparity of pixels that are not robust based on the disparity of the pixels and the correlation between the features of the pixels. Step 250 may be performed after step 240 .

In operation 251, the disparity determining device may obtain a correlation between disparity of a plurality of pixels. After determining the characteristics of each pixel, the disparity determination apparatus may obtain a correlation between characteristics of pixels of the left eye image and characteristics of pixels of the right eye image based on the characteristics of each pixel. The disparity determination apparatus may obtain a robustness value for each disparity by applying cross-validation to the disparity. The disparity determining apparatus may obtain a correlation between all disparity values based on the robustness value for each disparity.

As such, the disparity determination apparatus may obtain characteristics of each pixel of the binocular disparity image based on the distribution of sparse events and may obtain a correlation between the characteristics of pixels of the left eye image and the pixel characteristics of the right eye image. Accordingly, the parallax determination apparatus can effectively improve calculation speed by extracting features of each pixel using a relatively simple algorithm.

In operation 253, the disparity determination device may optimize the obtained disparity based on the correlation. The parallax determination device may optimize parallaxes using a dense conditional random field. The disparity determination device may determine disparity of matched pixels at a sub-pixel level based on the dense conditional random field.

The disparity determination device may obtain a dense conditional random field based on the robustness value and the correlation. The disparity determination device may obtain a dense conditional random field based on the disparity of the robust pixels and may indicate a correlation between events. The disparity determination device may determine sub-pixel level disparity by effectively predicting and optimizing the disparity of pixels that do not match based on the correlation between events and applying a smoothing filtering method to the disparity of adjacent pixels.

The disparity determination apparatus may obtain depth values of matched pixels based on a focal length of a camera that captures sub-pixel level disparity and binocular disparity images. The disparity determination device may optimize disparity based on depth values of pixels.

Since the binocular parallax image may include a left eye image and a right eye image, depth values of pixels may be applied to each image.

For example, for a left-eye DVS image, an angle in the left-eye DVS image is determined based on a camera focal length of the left-eye DVS image, a distance between the left-eye DVS camera and the right-eye DVS camera, and a parallax of each pixel in the left-eye DVS image at the sub-pixel level. A depth value of pixels may be obtained.

For example, with respect to the right eye DVS image, the angle in the right eye DVS image is determined based on the camera focal length of the right eye DVS image, the distance between the right eye DVS camera and the left eye DVS camera, and the parallax of each pixel in the right eye DVS image at the sub-pixel level. A depth value of pixels may be obtained.

For example, the disparity determination apparatus may obtain a robustness value for each disparity by applying cross-validation to the disparity using Equation 11.

와

가 같은지 여부를 판단한다. 같은 경우 픽셀 (x, y)은 강인한 픽셀로 판단되고, 다른 경우 픽셀 (x, y)은 강인하지 않은 픽셀로 판단된다. Here, the time difference determination device is

Wow

determine whether are equal. In the same case, pixel (x, y) is determined to be a robust pixel, and in other cases, pixel (x, y) is determined to be a non-robust pixel.

In order to predict the disparity of a non-robust pixel, the disparity determination device constructs a dense conditional random field E(D) by calculating the correlation between an arbitrary pixel and all other pixels, and optimizes the disparity of the pixel using this. can For example, the disparity determination device may optimize the disparity using Equation 12.

은 에너지 공식으로 각 픽셀의 시차의 강인성 값을 나타낸다. 교차 검증이 수행된 후에, 강인한 픽셀의 강인성 값은 N이고, 강인하지 않은 픽셀의 값은 0일 수 있다. 예를 들면 N=10일 수 있다.

은 에너지 공식으로 시차들 사이의 연관 관계를 나타낸다. 예를 들어, 그 정의는 수학식 13과 같을 수 있다.here,

represents the stiffness value of the parallax of each pixel with an energy formula. After cross-validation is performed, the robustness value of a robust pixel may be N, and the value of a non-robust pixel may be 0. For example, it may be N=10.

is an energy formula and represents a relation between parallaxes. For example, the definition may be as shown in Equation 13.

이고, 그러지 않으면

은 0이다. i, j는 픽셀의 식별 번호이고, w1과 w2는 가중치이며,

,

와

는 입력된 수치 파라미터다. Here, if di≠dj, then

is, otherwise

is 0. i and j are pixel identification numbers, w1 and w2 are weights,

,

Wow

is an input numerical parameter.

The disparity determination device may obtain a disparity at a sub-pixel level by optimizing disparity of pixels that are not robust based on a correlation between disparity of pixels. The optimization formula can be simplified by applying the gradient descent method. The simplified result is the parallax at the sub-pixel level of each pixel.

For example, the parallax determination device may calculate a depth value Z of each pixel based on a focal point f of the camera and a distance B of the camera. Here, Z = f*B/d, where d is the parallax at the sub-pixel level of a pixel.

Although each step has been described through specific equations above, the performance of each step is not limited to the above equations.

3 is a flowchart illustrating a procedure for acquiring features of pixels from a left-eye image and a right-eye image according to an exemplary embodiment.

According to one embodiment, at step 310, the parallax determination device may calibrate the DVS camera. Parameters such as the focal length of the left-eye DVS camera, the focal length of the right-eye DVS camera, and the distance between the left-eye DVS camera and the right-eye DVS camera may be calibrated.

According to an embodiment, in operation 320, the disparity determination apparatus acquires characteristics of a plurality of pixels included in the binocular disparity image based on the distribution of events of the binocular disparity image. Step 320 may include a process of extracting pixel features from the left eye image and a process of extracting pixel features from the right eye image. Each process may be performed simultaneously or sequentially.

The process of extracting pixel features from the left eye image includes acquiring a left eye image from the binocular parallax image (321), performing orthogonal decomposition on the left eye image to remove noise events (322), and removing noise events from the left eye image. A step 323 of performing Euclidean distance transform on the image and extracting pixel features may be included.

The process of extracting pixel features from the right eye image includes acquiring the right eye image from the binocular parallax image (321), performing orthogonal decomposition on the right eye image to remove noise events (322), and the right eye from which the noise events have been removed. A step 323 of performing Euclidean distance transform on the image and extracting pixel features may be included.

According to an embodiment, in operation 230, the disparity determination apparatus calculates a cost matrix of pixels matched between the left eye image and the right eye image based on the characteristics of the pixels.

According to an embodiment, in step 240, the disparity determination device determines a disparity of each matched pixel based on the cost matrix. In step 241, the disparity determination device may filter the cost matrix using a smoothing filtering method. In operation 243, the disparity determination device may determine a disparity of each matched pixel based on the filtered cost matrix. Then, in step 250, the disparity determination device may optimize the obtained disparity.

4 is a diagram visualizing features extracted according to a method for determining disparity of a binocular disparity image according to an exemplary embodiment.

4 is a visualization explanatory diagram in which pixel point features are extracted according to an embodiment of the present invention. The image 410 is an input DVS image, and may be a left eye image or a right eye image. The image 420 is a result of visualizing features extracted from pixels with respect to the image 410 .

5 is a diagram visualizing disparity obtained according to a method for determining disparity of a binocular disparity image according to an embodiment.

In FIG. 5 , an image 510 represents a binocular disparity image captured by an input DVS camera. A non-black part in the image 510 represents a part where an event has occurred. The image 520 is an image in which the disparity for each pixel acquired with respect to the image 510 is displayed as brightness. Brighter color in image 510 indicates greater parallax, indicating a closer distance from the camera.

6 is a diagram for comparing effects optimized according to an optimization method according to an embodiment.

The image 610 is an image visualizing the predicted disparity before optimization, and the drawing 620 is an image visualizing the predicted disparity after optimization. As can be seen in the image 610, the shape of the object photographed by the DVS camera cannot be maintained before the parallax is optimized, so the shape of the object is relatively greatly deformed. As can be seen in the image 620, after optimizing the parallax, the shape of the object photographed by the DVS camera is maintained and the shape of the object is relatively slightly deformed.

7 is a diagram showing the overall configuration of an apparatus for determining disparity of a binocular disparity image according to an exemplary embodiment. 8 is a flowchart illustrating an operation performed by each component of a device for determining a disparity of a binocular disparity image according to an exemplary embodiment.

According to an embodiment, the disparity determining device 700 includes a feature obtaining unit 720 , a matrix calculating unit 730 and a disparity determining unit 750 . According to another embodiment, the disparity determination apparatus 700 may further include a noise removal unit 710 , a filtering unit 740 and an optimization unit 760 .

The noise removal unit 710 may remove noise from the input binocular disparity image. The noise removal unit 710 may obtain a correlation between event distributions based on the degree to which event distributions are dense using orthogonal analysis. The noise removal unit 710 may represent the association relationship as a feature vector, and may remove noise from the binocular parallax image by determining as noise an event part corresponding to a feature vector having a small feature value of the feature vector.

The feature acquisition unit 720 acquires features of a plurality of pixels included in the binocular disparity image based on the distribution of events of the binocular disparity image. Also, the feature obtainer 720 may obtain features of a plurality of pixels included in the binocular disparity image based on the distribution of events of the binocular disparity image from which noise has been removed.

The feature acquisition unit 720 may acquire features of pixels using Euclidean distance transformation. The feature obtainer 720 may acquire pixel features by applying Euclidean distance transformation to each of the horizontal and vertical directions and combining the acquired Euclidean distances in each direction.

The matrix calculator 730 calculates a cost matrix of pixels matched between the left eye image and the right eye image based on the characteristics of the pixels. The matrix calculator 730 may obtain a feature matching cost of pixels matched between the left eye image and the right eye image by using feature matching. The matrix calculator 730 may obtain polarity matching costs of pixels matched between the left eye image and the right eye image by using polarity matching. The matrix calculator 730 may calculate a cost matrix based on a feature matching cost and a polarity matching cost.

The filtering unit 740 may filter the cost matrix. The filtering unit 740 may set a smoothing factor of the smoothing filtering method, and may filter the cost matrix by the smoothing filtering method based on the smoothing factor. Also, the filtering unit 740 may perform filtering using a Gaussian filtering method.

The disparity determiner 750 may determine a disparity of each matching pixel based on the filtered cost matrix. For example, the disparity determination unit 750 may determine the disparity of each matching pixel of the left eye image and the right eye image by using an algorithm such as a greedy winner takes all strategy.

The optimizer 760 may optimize the obtained parallax. The optimizer 760 may obtain a robustness value for each disparity by applying cross validation to the disparity. In order to predict the disparity of a pixel that is not robust, the optimizer 760 may construct a dense conditional random field by calculating an association between an arbitrary pixel and all other pixels, and optimize the disparity of a pixel using this. .

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Claims (16)

A method for determining parallax of a binocular parallax image,
acquiring characteristics of a plurality of pixels included in the binocular disparity image based on an event distribution of the binocular disparity image including a left eye image and a right eye image;
calculating a cost matrix of pixels matched between the left-eye image and the right-eye image based on the characteristics of the pixels; and
Determining a parallax of each of the matched pixels based on the cost matrix
including,
Calculating the cost matrix,
calculating a feature matching cost of pixels matched between the left eye image and the right eye image based on the feature of the pixels;
calculating a polarity matching cost of the matched pixels based on the polarities of the pixels; and
Obtaining a cost matrix of the matched pixels based on the feature matching cost and the polarity matching cost.
A method for determining disparity of a binocular disparity image comprising a.
According to claim 1,
The step of acquiring the feature is,
obtaining an event distribution by dividing a plurality of pixels included in the binocular parallax image into an event part and a non-event part;
calculating a Euclidean distance from each of the plurality of pixels included in the non-event portion to a nearest pixel included in the event portion; and
Setting each of the Euclidean distances as a feature of each of a plurality of pixels included in the non-event part.
A method for determining disparity of a binocular disparity image comprising a.
A method for determining parallax of a binocular parallax image,
acquiring characteristics of a plurality of pixels included in the binocular parallax image based on an event distribution of the binocular parallax image including a left eye image and a right eye image;
calculating a cost matrix of pixels matched between the left-eye image and the right-eye image based on the characteristics of the pixels; and
Determining a parallax of each of the matched pixels based on the cost matrix
including,
The step of acquiring the feature is,
obtaining an event distribution by dividing a plurality of pixels included in the binocular parallax image into an event part and a non-event part;
obtaining parabolas of a plurality of functions representing Euclidean distances between a plurality of pixels included in the non-event part and a plurality of pixels included in the event part;
obtaining points of intersection between the parabolas;
calculating a Euclidean distance to the nearest pixel based on the lower envelope of the intersection points; and
Setting each of the Euclidean distances as a feature of each of a plurality of pixels included in the non-event part.
A method for determining disparity of a binocular disparity image comprising a.
According to claim 3,
Calculating the cost matrix,
calculating a feature matching cost of pixels matched between the left eye image and the right eye image based on the feature of the pixels;
calculating a polarity matching cost of the matched pixels based on the polarities of the pixels; and
Obtaining a cost matrix of the matched pixels based on the feature matching cost and the polarity matching cost.
A method for determining disparity of a binocular disparity image comprising a.
According to claim 1,
The determining of the disparity may include filtering the cost matrix.
According to claim 1,
The method of determining disparity of a binocular disparity image, further comprising removing noise from the binocular disparity image.
A method for determining parallax of a binocular parallax image,
obtaining a plurality of feature vectors by applying orthogonal analysis to the binocular parallax image including a left eye image and a right eye image;
calculating feature values from the plurality of feature vectors;
removing noise from the binocular disparity image based on the feature values;
acquiring features of a plurality of pixels included in the binocular disparity image based on an event distribution of the binocular disparity image;
calculating a cost matrix of pixels matched between the left-eye image and the right-eye image based on the characteristics of the pixels; and
Determining a parallax of each of the matched pixels based on the cost matrix
A method for determining disparity of a binocular disparity image comprising a.
According to claim 1,
A method for determining disparity of a binocular disparity image, further comprising optimizing the determined disparity.
A method for determining parallax of a binocular parallax image,
acquiring characteristics of a plurality of pixels included in the binocular disparity image based on an event distribution of the binocular disparity image including a left eye image and a right eye image;
calculating a cost matrix of pixels matched between the left-eye image and the right-eye image based on the characteristics of the pixels;
determining a parallax of each of the matched pixels based on the cost matrix;
obtaining a correlation between parallaxes of the plurality of pixels; and
Optimizing the determined disparity based on the correlation
A method for determining disparity of a binocular disparity image comprising a.
According to claim 9,
The step of obtaining the association relationship is,
obtaining a robustness value for each of the lags by applying cross-validation to the lags; and
Obtaining a correlation between all lags based on the robustness value for each lag.
A method for determining disparity of a binocular disparity image comprising a.
According to claim 10,
The step of optimizing based on the association relationship,
obtaining a dense conditional random field based on the robustness value and the correlation; and
Optimizing the parallaxes using the dense conditional random field
A method for determining disparity of a binocular disparity image comprising a.
According to claim 11,
The step of optimizing using the dense conditional random field,
determining subpixel-level disparity of the matched pixels based on the dense conditional random field; and
Obtaining depth values of the matched pixels based on the sub-pixel level parallax and the focal length of a camera that captures the binocular parallax image.
A method for determining disparity of a binocular disparity image comprising a.
A non-transitory computer readable storage medium storing instructions that cause computing hardware to execute the method of any one of claims 1 to 12.
a feature acquiring unit that acquires characteristics of a plurality of pixels included in the binocular disparity image based on event distribution of the binocular disparity image including the left eye image and the right eye image;
a matrix calculation unit calculating a cost matrix of matching pixels between the left-eye image and the right-eye image based on the characteristics of the pixels; and
A disparity determination unit determining a disparity of each of the matched pixels based on the cost matrix.
including,
The matrix calculator,
Calculate a feature matching cost of pixels matched between the left eye image and the right eye image based on the feature of the pixels;
Calculate a polarity matching cost of the matched pixels based on the polarities of the pixels;
Obtaining a cost matrix of the matched pixels based on the feature matching cost and the polarity matching cost
A device for determining disparity in binocular disparity images.
A plurality of feature vectors are obtained by applying orthogonal analysis to a binocular parallax image including a left eye image and a right eye image, feature values are calculated from the plurality of feature vectors, and feature values are calculated in the binocular parallax image based on the feature values. a noise removal unit that removes noise;
a feature acquiring unit acquiring features of a plurality of pixels included in the binocular disparity image based on an event distribution of the binocular disparity image;
a matrix calculation unit calculating a cost matrix of matching pixels between the left-eye image and the right-eye image based on the characteristics of the pixels; and
A disparity determination unit determining a disparity of each of the matched pixels based on the cost matrix.
An apparatus for determining disparity of a binocular disparity image comprising a.
a feature acquiring unit that acquires characteristics of a plurality of pixels included in the binocular disparity image based on event distribution of the binocular disparity image including the left eye image and the right eye image;
a matrix calculation unit calculating a cost matrix of matching pixels between the left-eye image and the right-eye image based on the characteristics of the pixels;
a disparity determination unit determining a disparity of each of the matched pixels based on the cost matrix; and
An optimization unit that obtains a correlation between parallaxes of the plurality of pixels and optimizes the determined parallaxes based on the correlation.
An apparatus for determining disparity of a binocular disparity image comprising a.

Images