KR101807541B1 - Census pattern generating method for stereo matching - Google Patents

Abstract

The present invention provides a technique of generating a census pattern according to whether a difference in pixel values of a center pixel and a surrounding pixel belongs to a constant range in one window. A method for generating a census pattern for a stereo matching comprises the following steps of: an image processing device setting the window of a constant size in an image; and the image processing device determining a pattern value with respect to any one pixel according to whether the difference in a first pixel value which is a pixel value of the center pixel located at the center of the window and a second pixel value which is the pixel value of any one among the remaining pixels in the window is greater than or equal to a reference value.

Description

TECHNICAL FIELD [0001] The present invention relates to a method for generating a census pattern for stereo matching,

The techniques described below relate to techniques for generating census patterns for stereo matching.

In recent years, there is a great interest in technology for realizing human visual function using cameras in robot industry and unmanned automobile industry. Stereo Vision extracts three-dimensional information from images input by two cameras, just as humans recognize space using two eyes and cognitive abilities. Stereo matching uses binocular parallax to obtain depth information about how far an object is located.

The census transform, widely used in stereo matching algorithms, has a strong advantage in terms of image distortion and brightness variations. In the census conversion, the matching cost is determined according to the brightness value of the center pixel, and therefore, there is a characteristic of being sensitive to noise. Mini-census, generalized census, and the like have been proposed. The mini-census conversion is a method of comparing the brightness values of the central pixel by selecting six pixels without comparing all the brightness values of the surrounding pixels, and the generalized census conversion is a census conversion that is robust to Gaussian noise.

Martin Humenberger, "A Census-Based Stereo Vision Algorithm Using Modified Semi-Global Matching and Plane Fitting to Improve Matching Quality", 2010 IEEE Computer Society Conference on Vision and Pattern Recognition - Workshops

The technique described below is to provide a technique of generating a census pattern depending on whether a difference between a pixel value of a center pixel and a neighboring pixel in a window falls within a certain range.

There is provided a method of generating a census pattern for stereo matching, the method comprising: setting a window of a predetermined size in an image; and displaying the first pixel value, which is a pixel value of a center pixel positioned at the center of the window, Determining a pattern value for any one of the pixels based on whether a difference between the second pixel values, which is one of the remaining pixels, is equal to or greater than a reference value.

The techniques described below generate a census pattern that considers noise due to camera performance or shooting environment. This enables more accurate stereo matching.

1 is an example of a block diagram illustrating a configuration for a stereo image system.
Figure 2 is an example of a census transformation.
FIG. 3 shows an example of a change in pixel value of the same pixel in successive images.
4 is an example in which a census pattern is changed according to a change in a pixel value.
5 is an example of performing stereo matching using an incorrect census pattern.
FIG. 6 is an example of setting a constant range value for generating a census pattern.
7 is an example in which a reference value is set in a window.
8 is an example of a result of performing stereo matching using a census pattern.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include "should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms" comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner.

Also, in performing a method or an operation method, each of the processes constituting the above method may occur in a different order than that described in the context without explicitly specifying a specific order in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

The technique described below is to generate information for matching (matching) stereo images. Stereo vision technology is a technology that detects the same information on the same line in a binocular image input through two cameras and detects the distance information to the object by using difference of distance between them. The stereo matching method compares the information obtained by uniformly patterning the input image in the binocular image and detects matching information. A variety of techniques such as SAD, LBP, and Census can be used for information patterning for stereo matching. The technique described below relates to a census transformation technique.

FIG. 1 is an example of a block diagram showing the configuration of the stereo image system 100. Referring to FIG. The stereo image system 100 of FIG. 1 includes two cameras 110a and 110b, a video input device 120, an image processing device 130, and an output device 140. FIG.

In order to generate a stereo image, the first camera 110a and the second camera 110b basically acquire images of the same object or the same area. The image input device 120 is a device for simultaneously receiving images from two cameras 110a and 110b. The video input device 120 corresponds to a kind of signal interface device.

The image processing device 130 is a device for processing stereo images of two images (binocular images) transmitted from the image input device 120. For example, the image processing apparatus 130 may generate a depth map (dense disparity map) by stereo matching the binocular images. The image processing apparatus 130 may generate a census pattern for stereo matching.

The output device 140 is a device for outputting a video signal processed by the image processing device 130. For example, the output device 140 may be a computer monitor, a smart phone, a dedicated display device, or the like.

A process in which the image processing apparatus 130 generates a census pattern for stereo matching will be described. Figure 2 is an example of a census transformation. The image processing apparatus 130 sets a window 20 of a predetermined size in the image 10 and generates a census pattern in the set window 20. [ The image processing apparatus 130 may generate a census pattern for the entire image 10 while changing the window position in a predetermined order.

The image processing apparatus 130 generates a census pattern in units of pixels with respect to an area included in the window 20 of the image 10. [ An example of the census pattern is shown on the right side of FIG. The image processing apparatus 130 generates a census pattern based on a pixel located at the center of the window 20 (hereinafter referred to as a center pixel). In FIG. 1, the center pixel is indicated by 'x'. The image processing apparatus 130 compares the pixel value of the center pixel with the pixel value of any one of the pixels in the window. Hereinafter, a pixel to be compared with a center pixel and a pixel value is referred to as a target pixel.

If the pixel value of the target pixel in the window is greater than or equal to the pixel value of the central pixel, the image processing apparatus 130 allocates a pattern value of '1' to the target pixel. If the pixel value of the target pixel is smaller than the central pixel value, A pattern value of '0' is assigned to the pixel. Alternatively, if the pixel value of the target pixel in the window is greater than the pixel value of the central pixel, the image processing apparatus 130 may assign a pattern value of '1' to the target pixel. If the pixel value of the target pixel is less than the central pixel value, A pattern value of '0' may be assigned to a pixel. The pattern value (1 or 0) that the image processing apparatus assigns to the target pixel may use another value. The image processing apparatus 130 allocates a pattern value for the target pixel in the order of the arrows shown in Fig. 2 for the pixels existing in the window. In FIG. 2, when the pixel value of the target pixel is equal to or greater than the pixel value of the central pixel, the corresponding pixel is displayed as a shade. As a result, the image processing apparatus 130 generates a census pattern of "00100111 " for the pixels in the window 20. [ The census pattern is the result of comparing the center pixel with the remaining pixels except the center pixel.

The image processing apparatus 130 generates a census pattern for each of the binocular images and searches for the same information in the binocular image based on the generated census pattern.

On the other hand, the input image may not have the same value for the same pixel in consecutive frames depending on the change of light (illumination change) and / or hardware performance of the camera. FIG. 3 shows an example of a change in pixel value of the same pixel in successive images. 3 shows a change in the pixel value for a particular pixel 15 of the image 10. FIG. It is assumed that the actual background or object represented by the pixel 15 has no change in color or the like. The bottom of Fig. 3 represents the pixel value of the pixel 15 in successive frames. Referring to FIG. 3, it can be seen that the pixel value of the same pixel 15 changes in the image acquired by the camera. A normal pixel value indicated by a dotted line means a pixel value which the corresponding pixel 15 should originally display.

Since the census conversion determines the matching cost by the brightness value of the central pixel, the probability of false matching is greatly increased due to factors such as the noise of the center pixel brightness value. If the pixel 15 of FIG. 3 becomes a central pixel, it is difficult to trust the census conversion result performed by the image processing apparatus 130. Furthermore, even if the pixel values of other pixels included in the window change, the census pattern becomes inaccurate.

4 is an example in which a census pattern is changed according to a change in a pixel value. 4 (a) is an illustration of an example in which a census pattern is generated normally as shown in FIG. In Fig. 4 (a), the census pattern for the pixels included in the window 20 of the image 10 is "00100111 ". 4 (b) is an example of generating a census pattern for the window 20 at the same position as in Fig. 4 (a). It is assumed that the pixel values of two pixels (indicated by 'a1' and 'a2') of the peripheral pixels of the center pixel (indicated by 'x') have been changed. In this case, a census pattern different from that shown in Fig. 4 (a) is generated as shown in Fig. 4 (b). In Fig. 4 (b), the census pattern for the pixels included in the window 20 is "01000111 ".

If an incorrect census pattern is generated as shown in FIG. 4 (b), an error may also occur in stereo matching. 5 is an example of performing stereo matching using an incorrect census pattern. Stereo matching is performed using a left image and a right image of the binocular image. The image processing apparatus 130 performs stereo matching based on the census pattern of the binocular image. For example, the image processing apparatus 130 searches for a matching pattern on the same line (indicated by shading) in the left image based on the census pattern generated at the point p in the right image. Assume that the region coinciding with the census pattern generated at the p point of the right image in the left image is a census pattern generated at the point q.

It is assumed that a wrong census pattern is generated due to a change in pixel values in either the left image or the right image. In this case, as shown in FIG. 5, the image processing apparatus 130 can determine that the census pattern generated at the point q 'in the left image matches the census pattern generated at the point p in the right image have. As a result, the matching points of the left image and the right image based on the census pattern are determined differently. In this case, an inaccurate depth map can be generated.

The conventional census pattern was generated on the basis of the difference between the pixel value of the center pixel (hereinafter referred to as the first pixel value) and the pixel value of the target pixel (hereinafter referred to as the second pixel value). Therefore, when a pixel value change as shown in FIG. 3 occurs, a false census pattern can be generated. The technique described below determines that there is an actual difference only when the first pixel value and the second pixel value differ by a certain amount or more.

FIG. 6 is an example of setting a constant range value for generating a census pattern. As shown in FIG. 6, the image processing apparatus 130 regards pixel values included in a certain category (reference range) based on a normal pixel value as a value that does not differ from a normal pixel value. As a result, even if a slight pixel value change occurs, it is regarded as the same pixel value. If the difference between the first pixel value and the second pixel value is within the reference range, the image processing apparatus 130 determines the same pixel value to generate the census pattern. At least one pixel value of the first pixel value or the second pixel value may be changed into the surrounding environment or the device characteristic.

7 is an example in which a reference value is set in a window. 7 shows an example of a 5x5 window.

7 (a) is an example of an area for setting a reference value in the window 20. The reference value is a value for setting the reference range described in Fig. For example, if the reference value is 5, if the difference between the first pixel value and the second pixel value is within 5, the image processing apparatus 130 determines that the first pixel value and the second pixel value are the same pixel value. In FIG. 7 (a), the same reference value is used on the same rectangular line around the center pixel. For example, the image processing apparatus 130 may use the reference value 5 for the pixel located in the quadrangle 31, and use the reference value 3 for the pixel located in the quadrangle 32. [

7 (b) is another example of a region for setting a reference value in the window 20. In FIG. In Fig. 7 (b), the same reference value is used on the same concentric circle around the center pixel. For example, the image processing apparatus 130 may use the reference value 5 for the pixel located in the circle 41 and the reference value 3 for the pixel located in the circle 42. [

The image processing apparatus 130 can use a lower weight value as the target pixel moves away from the central pixel. On the other hand, the image processing apparatus 130 can determine the maximum value that can be used as the reference value based on the resolution of the pixel. For example, the image processing apparatus 130 can determine the maximum value of the reference value within 5% of the maximum pixel value representing the pixel. If the pixel value range is 0 ~ 255, the maximum value of the reference value can be 5. Of course, as an example, the image processing apparatus 130 may determine the reference value based on another criterion.

The image processing apparatus 130 may set a reference value based on the relative position of the target pixel with respect to the center pixel. The image processing apparatus 130 may set a reference value based on the distance between the center pixel and the target pixel. 7 (c) is another example of a region for setting a reference value in the window 20. In FIG. The image processing apparatus 130 sets the same reference value to the target pixel (denoted by A) having a pixel distance of 1 from the center pixel. The image processing apparatus 130 sets the same reference value to the target pixel (denoted by B) having a pixel distance of 2 from the center pixel.

FIG. 7 (d) is another example of a region for setting the reference value in the window 20. FIG. FIG. 7 (d) shows an example in which the distance between the center pixel and the target pixel is taken into account and the relative position of the target pixel with respect to the center pixel is considered. The image processing apparatus 130 may set the same reference value to the target pixel denoted by A. [ The image processing apparatus 130 may set the same reference value to the target pixel indicated by B. [ FIG. 7 (d) shows an example in which the same reference value is used for the rhombus-shaped region around the center pixel.

8 is an example of a result of performing stereo matching using a census pattern. 8A is a result of performing a stereo operation using a conventional census operation. 8 (b) is a result of performing a stereo operation through a census operation in which a reference range for a neighboring pixel of a center pixel and a center pixel is set to 3. 8 (c) is a result of performing a stereo operation through a census operation in which a reference range for a neighboring pixel of a center pixel and a center pixel is set to 5. FIG. It can be confirmed that a more stable stereo operation result can be obtained by applying a certain reference range.

It should be noted that the present embodiment and the drawings attached hereto are only a part of the technical idea included in the above-described technology, and those skilled in the art will readily understand the technical ideas included in the above- It is to be understood that both variations and specific embodiments which can be deduced are included in the scope of the above-mentioned technical scope.

100: stereo image system 100
110a: first camera
110b:
120: Video input device
130: Image processing device
140: Output device
10: Video
20: Windows

Claims (8)

Setting a window of a predetermined size in the image by the image processing apparatus; And
Comparing the difference between a first pixel value that is a pixel value of a center pixel positioned at the center of the window and a second pixel value that is a pixel value of one of the remaining pixels in the window to a reference value; as,
The reference value,
Wherein the first pixel value is determined based on a relative position of the one pixel with respect to the center pixel,
Determining that the second pixel value is different from the first pixel value if the difference exceeds the reference value and determining that the second pixel value is equal to the first pixel value if the difference is less than or equal to the reference value ; And
Determining a pattern value for any one of the pixels according to the determination result; , ≪ / RTI &
Wherein the image processing apparatus determines the pattern value for all pixels in the window to generate a census pattern. delete The method according to claim 1,
Wherein the reference value is reduced in proportion to a distance from the center pixel to any one of the pixels. The method according to claim 1,
Wherein the reference value is set to the same value for pixels at the same distance with respect to the center pixel. The method according to claim 1,
Wherein the reference value is set to the same value for pixels located on a concentric circle centered on the center pixel. The method according to claim 1,
Wherein the reference value is set to the same value for pixels at the same pixel distance with respect to the center pixel. The method according to claim 1,
Wherein the reference value has a value within a range of 5% of a maximum pixel value the pixel can have in the image. The method according to claim 1,
Wherein the reference value is set such that pixels having the same pixel distance as the center pixel are set to the same value and pixels having the same pixel value are set to have a lower value when the center pixel is farther from the center pixel.

Images