KR20110060600A - Apparatus and method for extracting location and velocity of obstacle - Google Patents

Abstract

PURPOSE: An obstacle location and velocity estimator and method are provided to improve the precision of the location and velocity estimation of an object with the application of an EKF. CONSTITUTION: A stereo matching unit(410) extracts a depth map from right and left images(400). An object detection unit(420) extracts an object. An integer parallax estimation unit(430) extracts integer parallax for a detected ROI(Region Of Interest). A precision parallax estimating unit(450) estimates precision parallax on a two-dimensional parabola. By the application of an EKF(Extended Kalman Filter), an EKF unit(480) estimates the location and velocity of the object on three-dimensional coordinates.

Description

Apparatus and Method for Extracting Location and velocity of Obstacle}

The present invention relates to an obstacle body position and speed estimation apparatus and method for estimating the position and speed of the obstacle body. More particularly, the present invention relates to an obstacle position and velocity estimation apparatus and method for estimating the position and velocity of an obstacle using stereo vision.

Obstacle Localization in three-dimensional space is one of the important problems in intelligent vehicle and robot navigation. This is especially important in the implementation of Adaptive Cruise Control (ACC) and Lane Keeping System (LKS), which are one of the essential functions in intelligent vehicles. In order to determine the location of obstacles, various sensors such as cameras, sonar, radar, and ray radar are used. Recently, data processing time has been reduced due to the improvement of computer performance and integration technology, and many researchers are using computer vision to perform obstacle positioning.

Computer vision has some problems with data processing time, but there are many advantages that are similar to human vision, where a lot of information is available from a single sensor. Recently, Stereo Vision (Stereo Vision) has been studied to overcome many problems of Mono Vision.

In the position measurement using a stereo camera, when the inverse transformation is performed on each of the left and right camera models, one point photographed in the left and right images is represented as one coordinate value on the global coordinate system. In this case, since the coordinate values expressed in the image are expressed in pixel units, a quantization error occurs. When the quantization error is converted into a distance, the error becomes larger as the distance increases and the distance precision decreases. In order to overcome this problem, a precise parallax estimation method using subpixel interpolation is used. However, in a real environment, stereo matching error and parallax estimation error occur due to a uniform region in an image, and thus there is a problem that distance accuracy and precision are degraded.

In order to solve the above problems, the present invention can improve the distance accuracy at a long distance, by applying an extended Kalman filter to estimate the object, the position of the obstacle object using a stereo camera that can minimize the position and speed error and It is an object of the present invention to provide a speed estimation apparatus and method.

In addition, the present invention can precisely estimate the parallax in sub-pixel units to reduce the range of parallax to be measured and the resolution required for stereo matching, which minimizes the computation time and cost required for estimating the distance of an object in the method of estimating integer parallax. Another object of the present invention is to provide an apparatus and method for estimating obstacle position and velocity using a stereo camera.

In order to achieve the above object, the present invention, an object to extract a depth map from the left and right images obtained by the stereo camera using a stereo matching algorithm, and to estimate the distance or speed using the depth map information Extract the integer parallax with respect to the detected region of interest (ROI), and the vertical edge components among the maximum dominant regions that are closest to the real object among the integer parallaxes. Extract only the pixels consisting of the feature points, extract stripers that are vectors in the line unit in which the feature points are distributed in the left and right images, and a normalized cross correlation vector composed of only feature points from the stripers. ), Calculate the center point of the left and right image by reflecting the estimated precision parallax in the right image, Obtain the position of the object in 3D coordinates using the center pixels of the two objects and Inverse Perspective Mapping (IPM), and dynamically estimate the position of the object in 3D coordinates using the Extended Kalman Filter. Provided are an obstacle speed and position estimation method and apparatus for optimizing and estimating a position and speed of a target.

As described above, according to the present invention, distance accuracy at a far distance can be improved, and an object can be estimated by applying an extended Kalman filter, thereby minimizing position and speed errors.

In addition, the present invention can precisely estimate the parallax in sub-pixel units to reduce the range of parallax to be measured and the resolution required for stereo matching, which minimizes the computation time and cost required for estimating the distance of an object in the method of estimating integer parallax. It can work.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible even though they are shown in different drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

In addition, in describing the component of this invention, terms, such as 1st, 2nd, A, B, (a), (b), can be used. These terms are only for distinguishing the components from other components, and the nature, order or order of the components are not limited by the terms. If a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected or connected to that other component, but between components It will be understood that may be "connected", "coupled" or "connected".

In the following description of an embodiment of the present invention, it is assumed that any object in the image is a road. That is, for example, the computer vision system mounted on a vehicle may photograph a road using two or more cameras and extract feature information of the road to detect an obstacle in an image of the photographed road. Explain. However, these assumptions are merely for convenience of explanation, and the present invention may be any object in the image, as well as roads, objects such as sidewalks, desks, indoors, playgrounds, or various body parts such as faces, cheeks, abdomen, and legs. It may be an object in the form.

1 is a relationship diagram of a stereo camera and a three-dimensional coordinate system.

As shown in FIG. 1, one point in the 3D global coordinate system is mapped to one pixel in the left and right images obtained in the stereo vision. If the camera model is inverted (IPM: Inverse Perspective Mapping) as shown in Equation 1 below, one point obtained from the left and right images can be obtained in a three-dimensional space.

In Equation 1, 3 denotes a 3D coordinate position in the global coordinate system, x denotes the x coordinate obtained from the left and right images, and y denotes the y coordinate obtained from the left and right images. Denotes a parallax obtained in stereo matching. Also, denotes a focal length in pixels, b denotes a distance between two cameras, and denotes an angle between the camera and the road.

Stereo matching is a problem of finding a corresponding point of another image from a reference point of one image, so that the epipolar line is matched to reduce the complexity, thereby converting the two-dimensional search into a one-dimensional search. In order to match the epipolar lines, preprocessing such as calibration and rectification is required, and as a result, the interisic and external parameters of the camera can be extracted.

When measuring the distance of a long obstacle, as in a vehicle, that is, it is mainly determined about the change of parallax, which means that the parallax error is converted into the error of the distance. At close distance, the error of distance with respect to parallax error is less affected, but at long distance, the distance error increases even with small error of parallax.

2 is a graph illustrating a relationship between parallax and distance.

In Equation 1, since it is expressed in units of pixels, a quantization error occurs, which causes a large distance error at a long distance. In order to solve this problem, the existing invention estimates the subpixel disparity in decimal units using a subpixel disparity estimation method based on normalized cross correlation (NCC). Using a subpixel parallax estimation method in a limited image (artificially generated image) may increase the accuracy (accuracy) than the pixel-based parallax estimated in stereo matching. However, in a real image, matching errors and parallax estimation errors occur in a homogeneous region, thereby degrading accuracy and precision (see FIG. 3).

The present invention proposes a feature-based precision parallax estimation method that can minimize matching error and parallax estimation error, and by applying the Extended Kalman Filter (EKF), it is possible to improve the precision and estimate the position and velocity of the object. It provides an apparatus and method that can be.

The onset is based on the method of estimating the precise parallax using a stereo camera and estimating the position of the obstacle, and by applying the extended Kalman filter (EKF) based on the estimated parallax, Provided are an apparatus and method for estimating.

4 is a block diagram of an obstacle position and velocity estimation apparatus according to an embodiment of the present invention.

Obstacle velocity and position estimation apparatus according to an embodiment of the present invention is a stereo matching unit 410 and the object detector 420, integer parallax estimator 430, precision parallax estimator 450, object center estimator ( 460, a PMI unit 470, and an extended knife only filter unit 480.

The first image capturing apparatus and the second image capturing apparatus not shown in FIG. 4 respectively photograph left and right images necessary for estimating distance, speed, etc. in the stereo vision system. As the image photographing apparatus, for example, a camera or the like may be used. The left image and the right image 400 obtained by the first image capturing apparatus and the second image capturing apparatus are input to the stereo matching unit 410.

The stereo matching unit 410 extracts a depth map from the left and right images 400 obtained by the stereo camera using a stereo matching algorithm.

The object detector 730 may extract an object for estimating a distance or a speed using depth map information. The object may be an obstacle on the road. This obstruction can be detected using algorithms such as V-disparity, column detection, adamboost, support vector machine (SVM), and the like.

The integer parallax estimator 430 extracts an integer parallax with respect to the detected region of interest (ROI). When the integer parallax is extracted, various parallax values may exist in the ROI according to the shape of the object and the background component. The integer parallax estimator 430 extracts the most dominant of these parallax values and the max dominant region located at the closest distance.

5 is a flowchart illustrating a method of extracting an area of the nearest distance from the integer parallax estimator of FIG. 4.

As shown in FIG. 5, the most important region (a green circular display portion in the graph) is extracted from the ROI acquired from the original image. The most important region, that is, the region closest to the object (the red circular display portion in the graph) is extracted. Only pixels consisting of vertical edge components in the nearest dominant region are extracted as feature points. In the left and right images, the feature extracts a line-by-line vector, that is, stripers, and generates a normalized cross correlation vector (NCC vector) composed of only feature points from the stripers. Normal similarity (NCC) represents a normalized value of the similarity degree of pixels in a window.

6 is a diagram illustrating feature points and object center extraction.

The precision parallax estimator 450 estimates the precision parallax of a subpixel unit on a two-dimensional parabola using the similarity of the generated NCC vectors.

For example, an NCC value of a window set with a center pixel of a left image as a center pixel and a window set with a corresponding pixel as a center pixel in a right image is referred to as NCC (k1), and the center of an object of a left image as a center pixel. The NCC value of the window set by using the shifted pixel to the right from the corresponding pixel of the right window and the set pixel as the center pixel is NCC (k1-1), and the window set by using the center of the object of the left image as the center pixel. And the NCC value of the window set with the pixel shifted one pixel to the left from the corresponding pixel of the right image as a center pixel is referred to as NCC (k1 + 1). Equations 2 and 3 below represent equations for obtaining the NCC.

In Equation 2 and Equation 3, W (x, y) is a brightness value at x, y coordinates in a window, and Wm (x, y) is an average value of luminance of all pixels in the window, W '(x, y). ) Represents a value obtained by dividing the luminance value in the x, y coordinates by subtracting the average value of the luminance for all pixels by the norm value. In Equation 3, the operator 크로스 cross-correlates the W _L '(x, y) value of each pixel in the window of the left image and the W _R ' (x, y) value of each pixel in the window of the right image. Represents a cross correlation operation. d ₀ represents a parallax value corresponding to the closest maximum region obtained by the integer parallax estimator. k indicates the amount of shift of the center pixel of the window when the NCC value is obtained by shifting the position of the window. In the above example, the case in which the window of the right image is moved by using the left image as a reference image is taken as an example. However, when the right image is the reference image according to the embodiment as described above, the window of the left image is moved. You can also implement

Using the three obtained NCC values (NCC (k1-1), NCC (k1), NCC (k1 + 1)), the precise parallax at which the normal similarity is maximized can be estimated. For example, assume that the three NCC values form a secondary parabola.

FIG. 7 illustrates a subpixel parallax estimation method when three NCC values form a left-right symmetric secondary parabola.

If the difference between the center pixel of the center pixel of the left image and the right image window in the window k1 = d _int value NCC _(int d), the center of the pixel difference d _int +1 value NCC (d _int +1), the center If the pixel difference is d _int -1, it can be seen that the NCC (d _int -1) value appears in the quadratic parabolic coordinate. In FIG. 7, it can be seen that there is a difference between the parallax d _acc at the point where the normal similarity is maximum and the parallax d _int of the left and right window center pixels. This is due to the inter-pixel quantization error in pixel-based stereo vision systems. The subpixel parallax d _acc at the point where the normal similarity (NCC) is maximized can be obtained using the ^second parabola (y = ax ² + bx + c) and three NCC values.

At this time, the number of stripers can be made N, and using the average of N, to estimate the precise parallax of the object.

The center point of the object in the left and right images is obtained by reflecting the estimated subpixel parallax (d _acc ), that is, the precise parallax of the object in the right image.

The IPM unit 470 may obtain the position of the object in 3D coordinates using the estimated center pixels of the two objects and the inverse perspective mapping (IPM) described above.

The extended Kalman filter unit 480 may estimate the position and velocity of the object by dynamically estimating the speed and the position of the object on the 3D coordinates dynamically using the Extended Kalman filter.

8a to 8b show the results of experiments on the actual road for the position and speed using the precision parallax and IPM-based Extended Kalman filter.

9 is a flowchart of a method for estimating obstacle speed and position according to another embodiment of the present invention.

Obstacle velocity and position estimation method according to another embodiment of the present invention is a stereo matching step (910) and depth map information for extracting a depth map (Depth map) using a stereo matching algorithm from the left and right images obtained by a stereo camera An object detection step 920 for extracting an object for which distance or velocity is to be estimated, an integer parallax estimating step 930 for extracting integer parallax with respect to the detected ROI, and the most important of these parallax values Extracts the dominant region (max dominant region), extracts the vertical edge components within the region of interest (ROI) obtained from the original image, and most importantly, the maximum dominant region Extract only the pixels consisting of the vertical edge components into the feature points, and in the left and right images, A tri-flops of the (stripes) extraction, the normal degree of similarity consisting of only the feature points from the Stryper (Normalized Cross Correlation), the vector (NCC vector) Fine Time estimate step 950, the estimated sub-pixel difference (d _acc) to generate The object center estimation step 960 of calculating the center point of the object of the left and right image reflected on the right image, the position of the object on the three-dimensional coordinates using the estimated center pixels of the two objects and the inverse perspective mapping (IPM) described above. IPM step 970 of obtaining the extended Kalman filter step 980 to estimate the position and velocity of the object by dynamically estimating the position and velocity of the object dynamically using the Extended Kalman Filter. Include.

Detailed description of each step is the same as the description of the respective components of the obstacle velocity and position estimation apparatus described with reference to FIGS.

Obstacle object position and speed estimation apparatus and method using a stereo camera according to embodiments of the present invention can improve the distance accuracy at a long distance, by applying the extended Kalman filter to estimate the object and the position, speed and position The error can be minimized.

By accurately estimating the parallaxes in subpixel units using an apparatus and a method for estimating the obstacle object position and the speed using a stereo camera according to embodiments of the present invention, the range of parallax to be measured and the resolution required for stereo matching can be reduced. This can minimize the computation time and cost required to estimate the distance of an object in the integer parallax estimation method.

The present invention can be applied to technologies such as accident reduction system, driving assistance system, autonomous driving using a stereo vision system in an intelligent vehicle.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. In other words, within the scope of the present invention, all of the components may be selectively operated in combination with one or more. In addition, although all of the components may be implemented in one independent hardware, each or all of the components may be selectively combined to perform some or all functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. Codes and code segments constituting the computer program may be easily inferred by those skilled in the art. Such a computer program may be stored in a computer readable storage medium and read and executed by a computer, thereby implementing embodiments of the present invention. The storage medium of the computer program may include a magnetic recording medium, an optical recording medium, a carrier wave medium, and the like.

In addition, the terms "comprise", "comprise" or "having" described above mean that the corresponding component may be inherent unless specifically stated otherwise, and thus excludes other components. It should be construed that it may further include other components instead. All terms, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. Terms used generally, such as terms defined in a dictionary, should be interpreted to coincide with the contextual meaning of the related art, and shall not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present invention.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

1 is a relationship diagram of a stereo camera and a three-dimensional coordinate system.

2 is a graph illustrating a relationship between parallax and distance.

3 is a diagram illustrating a matching error and a parallax estimation error.

4 is a block diagram of the obstacle speed and position estimation apparatus according to an embodiment of the present invention.

5 is a flowchart of an integer parallax estimation method of the integer parallax estimator of FIG. 4.

6 is a diagram illustrating feature points and object center extraction.

FIG. 7 illustrates a subpixel parallax estimation method when three NCC values form a left-right symmetric secondary parabola.

8a to 8b show the results of experiments on the actual road for the position and speed using the precision parallax and IPM-based Extended Kalman filter.

9 is a flowchart of a method for estimating obstacle speed and position according to another embodiment of the present invention.

Claims (6)

A stereo matching step of extracting a depth map from a left and right image obtained by a stereo camera using a stereo matching algorithm; An object detecting step of extracting an object for estimating a distance or a speed using depth map information; An integer parallax estimating step of extracting integer parallax with respect to the detected ROI; A line in which only the pixels composed of vertical edge components among the max dominant regions corresponding to a real object among the integer parallaxes are extracted as feature points, and the feature points are distributed in left and right images. A precision parallax estimating step of extracting stripers, which are vectors of units, and generating a normalized cross correlation vector (NCC vector) consisting of only feature points from the stripers; An object center estimation step of obtaining a center point of an object in left and right images by reflecting the estimated precision parallax in the right image; An IPM step of obtaining the position of the object in three-dimensional coordinates by using the estimated center pixels of the two objects and inverse perspective mapping (IPM); And, An obstacle velocity and position estimation method comprising an extended Kalman filter step of optimizing the position and velocity of an object by dynamically estimating the position and velocity of an object in three-dimensional coordinates using an extended Kalman filter. The method of claim 1, In the object detection step, the distance or velocity using depth map information using one of the following algorithms: V-disparity, column detection, adabost, and SVM (support vector machine). Obstacle velocity and position estimation method, characterized in that for extracting the object to be estimated. The method according to claim 1 or 2, In the precise parallax estimating step, Among the integer parallaxes, a maximum dominant region is extracted, and a maximum dominant region within the ROI obtained from the original image is extracted. A method for estimating obstruction speed and position, comprising extracting only pixels consisting of vertical edge components as feature points in. A stereo matching unit which extracts a depth map from a left and right image obtained by a stereo camera using a stereo matching algorithm; An object detector for extracting an object for estimating a distance or a speed using depth map information; An integer disparity estimator extracting an integer disparity with respect to the detected ROI; A line in which only the pixels composed of vertical edge components among the max dominant regions corresponding to a real object among the integer parallaxes are extracted as feature points, and the feature points are distributed in left and right images. A precision parallax estimator extracting stripers that are vectors of units, and generating a normalized cross correlation vector (NCC vector) consisting of only feature points from the stripers; An object center estimating unit for calculating the center point of the object in the left and right images by reflecting the estimated precision parallax in the right image; An IPM unit which obtains a position of an object in three-dimensional coordinates by using estimated center pixels of the two objects and inverse perspective mapping (IPM); And, Obstacle velocity and position estimating apparatus comprising an extended Kalman filter unit for optimizing the position and velocity of the object by dynamically estimating the position and velocity of the object in the three-dimensional coordinates using the extended Kalman filter. The method of claim 4, wherein In the object detection unit, a distance or a speed may be determined using depth map information using one of an algorithm such as V-disparity, column detection, adabost, and SVM (support vector machine). Obstacle velocity and position estimation apparatus, characterized in that for extracting the object to be estimated. The method according to claim 4 or 5, The precise parallax estimator extracts a maximum dominant region located at the closest dominant distance among the integer parallaxes, and extracts the most significant and closest region within the ROI obtained from the original image. 2. A obstacle velocity and position estimating apparatus, characterized in that only pixels consisting of vertical edge components are extracted as feature points within a max dominant region.

Images