JP3870255B2 - Three-dimensional restoration method and apparatus - Google Patents

Description

で標準ステレオカメラ系の座標（Ｘ，Ｙ）に変換される。また、焦点距離ｆは、変換後の画像サイズが元の画像のサイズにほぼ同一になるように定める。以後、画像の座標に関しては、標準カメラモデル座標系の座標値に変換したものを扱う。例えば、図３のステレオ画像に対して、標準ステレオカメラモデル系に変換を施した画像は図４となる。この変換により、４角形の頂点は変換前の図３では基準・参照画像ではｒｏｗ値は一致していないが、変換後の図４では一致するようになる。
【００１６】
また、ステレオにおいては、エピポーラ線と平行なセグメントは対応関係が不定になりやすいため、ステップＳ２で求めたセグメントの構成する点列のＹ値から、水平セグメントと非水平セグメントに分割し、さらに非水平セグメントに対して、極大／極小点で再分割する。これらの処理により、非水平セグメントにおいては、構成する点列はＹ値について見ると単調増加（減少）が保証される。
【００１７】
（５）セグメント対応（Ｓ５）
ステップＳ４で求めたセグメントを対応単位として対応を求める。まず、エピポーラ条件を満たすセグメントを選択し、形状、方向、輝度などの特徴が類似しているセグメントを対応候補として求める。次に、求めた対応候補同志の連結性を調べ、連結性を基に、基準画像の境界線と類似したセグメントの集まりを参照画像中から求める処理を行い、その類似度を計算する。そして、求めた類似度を評価基準として、選ばれた対応候補の中から対応が重なり合わない、つまり、対応関係が一対一になる正しい対応を求める。
【００１８】
（６）対応点計算（Ｓ６）
次に、対応セグメント上の各点の対応点を求める。対応点はエピポーラ条件を満足するので、図２の標準カメラモデル座標系の基準・参照画像ではそれぞれ（ＸL，Ｙ），（ＸR，Ｙ）となる。ただし、実際の処理においてはセグメンテーションは原画像で行い、点の座標値のみを標準カメラモデル座標系に変換している。例えば、図３の原画像を図４のような標準カメラモデル座標系に基づく画像に変換すると、画像全体を変換するため処理時間がかかる点や、画像が平滑化されてしまう欠点がある。対象点の座標値のみの変換をすることで、これらの欠点を克服できる。しかしながら、点の座標値のみの変換のため、図５に示すように、基準画像のセグメントを構成する各白丸点○のＹ値は、参照画像のセグメント上の点のＹ値と通常は一致しない。ステップＳ４において、非水平セグメントにおけるＹ値は単調増加（減少）を保証するように分割しているので、Ｙ を挟み込む連続する２つの黒丸点●（ＸR1，ＹR1），（ＸR2，ＹR2）を結んだ直線がＹとなる点×（ＸR，Ｙ）を計算し、この点を（ＸL，Ｙ）の対応点とする。水平セグメントに関してはＹは一定となるため、この処理は必要ない。図３の４角形の４辺部分の対応点間の標準カメラモデル座標系における視差は図６であり、対象が直線であるにもかかわらずがたつきがある。
【００１９】
（７）平面拘束条件計算（Ｓ７）
セグメント間の対応を求める方法は連結性に基づいた手法であるため、隣接セグメントとの連結性を容易に得ることができる。故に、対象セグメントと連結セグメントの組合わせで求まる平面に点列が存在する場合は、その平面上に投影することで、法線方向の誤差を軽減できる。対象セグメントと連結セグメントが平面上に存在するかどうかは、平面拘束定理「標準カメラモデルにおいて、平面上の任意の図形の一方のカメラ画像上の投影像と、他方のカメラの画像上の投影像とはアフィン変換可能である。」により調べることができる。この平面拘束により、対応点の補正が可能で、また、従来の手法において、連結すべきセグメントが３次元的にねじれの位置に復元される問題も解決できる。
【００２０】
図７に示すような、対応する基準、参照画像のセグメントＳLi，ＳRjに属する対応点（点数：ｎ）を（ＸLik，Ｙ），（ＸRjk，Ｙ）とし、それらが平面上に存在するならば、平面拘束定理から、
【数２】

というアフィン変換が成立する。各セグメントに対して、以下のアルゴリズムで連結するセグメントの情報を使用して平面拘束条件、つまり、アフィン変換行列Ａを計算する。
【００２１】
（ａ）対象セグメントの組（ＳLi，ＳRj）に対し、連結する前後のセグメントの組（ＳLi-1，ＳRj-1），（ＳLi+1，ＳRj+1）の存在を調べる。
【００２２】
（ｂ）対象セグメントと連結するセグメントが同一直線上に存在する場合は、それらを１つの対象セグメントの組（ＳLi’，ＳRj’）と見なして、（ａ）に戻り連結性を調べ直す。
【００２３】
（ｃ）選択したセグメントの組の対応点から、「数３」のεを最小にする行列Ａを求める。
【数３】

ただし、連結性がなく（ＳLi，ＳRj）だけの場合、それが曲線セグメントならば単独で平面を定義できるが、直線セグメントの場合はａ＝１として、視差をＹの１次関数で表現、つまり直線フィットを行う。
【００２４】
（ｄ）誤差Ｅを「数４」で定義した場合、Ｅがしきい値以下であれば、「数３」を満足するとしてステップＳ８の対応点修正に進む。それ以外は、そのセグメントの組合せでは平面が構成されないとし、連結するセグメントの組で、Ｅが大きくなる原因の組の方を除いて（ｃ）に戻る。対象セグメントの組だけになった場合はステップＳ８の対応点修正に進む。
【数４】

【００２５】
（８）対応点修正（Ｓ８）
ステップＳ７で平面拘束条件が成立するための行列Ａが求まったので、平面拘束条件を満たすように対応点修正を行う。具体的には、対象セグメントの参照画像の対応点のＸRを「数２」を利用してＸR’に変換する。
【数５】

【００２６】
（９）３次元復元（Ｓ９）
対象セグメントの（ＸL，Ｙ）と、補正した対応点（ＸR’，Ｙ）が求まるので、標準ステレオカメラモデルでの基線長をｂ、焦点距離をｆ、観測点の３次元座標値を（ｘ，ｙ，ｚ）とすると、この２点は
【数６】

と表すことができ、これを変形すると観測点は
【数７】

から計算できる。これにより求めた点列は平面拘束からセグメントの組合わせによって決まる平面上に投影されるので、その法線方向に対する誤差が軽減される。
【００２７】
次に、平面上での誤差を軽減する処理を行う。対象が直線セグメントの場合は、各画像で共にエピポーラ線に平行でない（Ｙ≠一定）直線に投影されているならば、３次元的にも直線であることから、その平面上に投影された点列に対して直線を当てはめ、３次元空間において直線になるように変換を行う。曲線においても同様に平面上で曲線フィットを行う。
【００２８】
図８は図２の図形に対し、ステップ６で求めた対応点の座標値を「数６」に適用して得られた３次元形状復元結果を左斜め上方の視点から観測した結果であり、図６の視差グラフで示したように、３次元的な、特に奥行き方向に、がたつきが見られる。一方、本発明を適用した結果を同視点から観測したものが図９である。連結性を考慮して平面拘束した直線当てはめになるため、隣接するセグメントとはねじれの位置関係になることなく、より高精度に４角形が復元されている。
【００２９】
【発明の効果】
以上のように、本発明によれば、ステレオ画像において、セグメントの連結性に基づく平面拘束から視差を修正することができ、３次元幾何形状を高精度に復元できるという効果があり、産業用視覚システムやロボットに視覚を与えるために利用することができる。
【図面の簡単な説明】
【図１】本発明に係る３次元復元を示す工程図である。
【図２】標準ステレオカメラモデルを示す図である。
【図３】原画像を示す図である。
【図４】標準ステレオカメラモデル系に変換を施した画像を示す図である。
【図５】対応点を示す図である。
【図６】対応点間の視差を示す図である。
【図７】連結する前後のセグメントの組を示す図である。
【図８】対応点間の視差から３次元幾何形状復元した結果を示す図である。
【図９】本発明の手法を適用し、３次元幾何形状復元した結果を示す図である。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method and apparatus for correcting parallax from plane constraints based on segment connectivity in stereo images and restoring a three-dimensional geometric shape with high accuracy.
[0002]
[Prior art]
The present applicant has previously proposed a stereo image luminance correction technique useful for three-dimensional geometric shape restoration (see Japanese Patent No. 2958462). The method of restoring the three-dimensional geometric shape existing in the scene from the stereo image is to search the correspondence of each pixel between the images when the relative positional relationship of the camera at the time of shooting is known, and to perform the three-dimensional analysis by the principle of triangulation It is to restore information. In order to restore the three-dimensional information, it can be calculated from the parallax between corresponding points. For example, the following method is known.
[0003]
(1) First, there is a method of simply restoring from parallax only. An error in parallax caused by an error at the time of edge detection greatly affects the restoration accuracy, resulting in a large three-dimensional rattling.
[0004]
(2) Second, there is a technique of smoothing by using neighboring points to reduce errors. However, this does not mean that there will be no significant rattling and it will not be an essential solution.
[0005]
(3) Thirdly, a straight line or a curve is applied to each corresponding segment by the least square method or the like, thereby eliminating shakiness of the points constituting the segment. This method eliminates blurring of the point cloud in the segment, but it is an improvement for each segment, and the position of the apex portion is shifted, so there are many cases in which the positional relationship between the adjacent segment and the torsion is 3 The continuity of the dimension position is not guaranteed.
[0006]
(4) Fourth, there is a method of connecting vertices so as not to have such a twisted positional relationship. This method satisfies the epipolar condition, that is, the continuity of the three-dimensional position at the vertex is guaranteed, but the point detection instability in the edge detection process and the information of the intermediate part are not utilized. There are problems such as points.
[0007]
[Problems to be solved by the invention]
With the conventional method of restoring the three-dimensional geometric shape, it is difficult to restore with high accuracy.
In view of the above points, the present invention has been made by paying attention to the above-described problem that the reconstruction accuracy of the three-dimensional geometric shape is not good. The object is to correct the corresponding points so as to satisfy the condition and restore the three-dimensional geometric shape with high accuracy.
[0008]
[Means for Solving the Problems]
The three-dimensional high-accuracy restoration method according to the present invention includes a step of detecting an edge in a stereo image and dividing the segment into segments by a local shape to obtain a segment feature amount, and based on the feature amount, the segment is defined as a corresponding unit. A step of obtaining a correspondence between stereo images by performing a stereo correspondence search process, a step of converting a coordinate system to make it easy to obtain the correspondence in a stereo image, and calculating a corresponding point from the corresponding portion based on the correspondence A step of obtaining a plane constraint condition from a segment connection relationship based on the correspondence relationship, a step of correcting a corresponding point to satisfy the plane constraint condition in the stereo image, and a relationship of the corresponding points in the stereo image When restoring the 3D shape from the 3D shape It is obtained so as to have a step of performing fit.
[0009]
Also, the three-dimensional high-accuracy restoration device according to the present invention corresponds to a means for detecting an edge in a stereo image, dividing the segment into segments by a local shape, and obtaining a segment feature amount, and corresponding the segment based on the feature amount. A means for obtaining a correspondence between stereo images by a stereo correspondence search process as a unit, a means for converting a coordinate system to make it easy to obtain the correspondence in a stereo image, and a corresponding point from a corresponding portion based on the correspondence Means for calculating a plane constraint condition based on the connection relation of the segments based on the correspondence relationship, means for correcting the corresponding point so as to satisfy the plane constraint condition in the stereo image, and a corresponding point in the stereo image The means for restoring the 3D shape from the relationship between Those having means for performing fitting.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described based on examples. FIG. 1 is a process diagram showing a three-dimensional restoration process. The three-dimensional restoration method and apparatus according to the present invention detects a edge in a stereo image, divides the segment into segments by a local shape, obtains a segment feature amount, and uses the segment as a corresponding unit based on the feature amount. Correspondence between stereo images is obtained by correspondence search processing, the coordinate system is converted to make the correspondence easier to find, the corresponding points are calculated from the corresponding portions, and based on the correspondence, the plane constraint condition is determined from the segment connection In the stereo image, the corresponding points are corrected so as to satisfy the plane constraint condition, the three-dimensional shape is restored from the relationship of the corresponding points, and straight / curve fitting is performed according to the segment attributes.
[0011]
Hereinafter, a three-dimensional high-accuracy restoration process in a stereo image will be described step by step with reference to FIGS.
(1) Edge detection (S1)
In each image, an edge is detected at a portion where the luminance change is large. For example, the product sum of the Sobel operator is calculated at each point, the non-maximum value is erased, and then the edge portion is extracted by performing an extended image process according to the result compared with the threshold value. The edge here is a collection of pixel points, that is, a sequence of points.
[0012]
(2) Segment extraction (S2)
The detected edge is divided into segments at feature points such as branch points, refraction points, inflection points, and transition points according to the local shape. A line segment constituted by a collection of points between the feature points is called a segment.
[0013]
(3) Segment information acquisition (S3)
Each segment is constituted by a point sequence that is a collection of points, and the feature amount of each segment such as the shape, length, direction, and luminance of the segment is calculated.
[0014]
(4) Standard camera model coordinate system conversion (S4)
In the stereo image, in order to simplify the correspondence calculation in the reference image, the photographed image is virtually converted into a standard stereo camera model system image as shown in FIG. 2 from the camera calibration data calculated in advance. . The standard stereo camera model is a camera model in which the focal centers of the reference and reference cameras are OL and OR, respectively, and the image plane exists at a distance f (focal length) in the direction perpendicular to the line segment OLOR. In this model, the projection points of the observation point P on the standard / reference image are PL and PR, respectively. That is, the point PR on the reference image corresponding to the point PL observed on the standard image exists on a straight line where the plane obtained by the three points OL, OR, and P (PL) intersects the reference image plane. . This is called an epipolar line, and the corresponding point search need only search on the epipolar line, not the entire image. This is called an epipolar condition. In this model, the epipolar condition is such that the Y coordinate value of a point on the image plane is constant, and by performing this conversion, it is easy to search for correspondence in a stereo image.
[0015]
As a result, each point (col, row) on the image belonging to the segment is
[Expression 1]

Is converted into the coordinates (X, Y) of the standard stereo camera system. The focal length f is determined so that the image size after conversion is substantially the same as the original image size. Hereinafter, regarding the coordinates of the image, those converted into the coordinate values of the standard camera model coordinate system are handled. For example, an image obtained by converting the stereo image of FIG. 3 into a standard stereo camera model system is shown in FIG. With this conversion, the vertex values of the quadrangles do not match in the row values in the standard / reference image in FIG. 3 before conversion, but in FIG. 4 after conversion.
[0016]
In stereo, the segment parallel to the epipolar line is likely to have an indefinite correspondence, so the Y value of the point sequence formed by the segment obtained in step S2 is divided into a horizontal segment and a non-horizontal segment. For a horizontal segment, subdivide by maximum / minimum points. By these processes, in the non-horizontal segment, the constituent dot sequence is guaranteed to be monotonously increased (decreased) in terms of the Y value.
[0017]
(5) Segment support (S5)
The correspondence is obtained using the segment obtained in step S4 as a corresponding unit. First, a segment that satisfies the epipolar condition is selected, and a segment having similar characteristics such as shape, direction, and luminance is obtained as a corresponding candidate. Next, the connectivity of the obtained correspondence candidates is examined, and based on the connectivity, a process for obtaining a collection of segments similar to the boundary line of the reference image from the reference image is performed, and the similarity is calculated. Then, by using the obtained similarity as an evaluation criterion, a correct correspondence in which the correspondence does not overlap among the selected correspondence candidates, that is, the correspondence relationship becomes one-to-one is obtained.
[0018]
(6) Corresponding point calculation (S6)
Next, the corresponding point of each point on the corresponding segment is obtained. Since the corresponding points satisfy the epipolar condition, they are (XL, Y) and (XR, Y) in the standard / reference image of the standard camera model coordinate system of FIG. However, in actual processing, segmentation is performed on the original image, and only the coordinate values of the points are converted into the standard camera model coordinate system. For example, when the original image of FIG. 3 is converted into an image based on the standard camera model coordinate system as shown in FIG. 4, it takes a processing time to convert the entire image, and the image is smoothed. By converting only the coordinate value of the target point, these drawbacks can be overcome. However, because only the coordinate value of the point is converted, as shown in FIG. 5, the Y value of each white circle point ○ constituting the segment of the standard image does not normally match the Y value of the point on the segment of the reference image. . In step S4, the Y value in the non-horizontal segment is divided so as to guarantee a monotonous increase (decrease). The point x (XR, Y) where the straight line becomes Y is calculated, and this point is set as the corresponding point of (XL, Y). Since Y is constant for the horizontal segment, this processing is not necessary. The parallax in the standard camera model coordinate system between corresponding points of the four sides of the quadrangle in FIG. 3 is FIG. 6, and there is shakiness even though the target is a straight line.
[0019]
(7) Plane constraint condition calculation (S7)
Since the method for determining the correspondence between segments is a technique based on connectivity, connectivity with adjacent segments can be easily obtained. Therefore, when a point sequence exists on a plane obtained by a combination of the target segment and the connected segment, an error in the normal direction can be reduced by projecting onto the plane. Whether the target segment and the connected segment exist on the plane is determined by the plane constraint theorem "In the standard camera model, the projected image on one camera image of any figure on the plane and the projected image on the other camera image Can be affine transformed. " By this plane constraint, the corresponding points can be corrected, and the problem that the segments to be connected are three-dimensionally restored to the twisted position in the conventional method can be solved.
[0020]
As shown in FIG. 7, if the corresponding points (score: n) belonging to the corresponding standard and reference image segments SLi, SRj are (XLik, Y), (XRjk, Y), and they exist on the plane, From the plane constraint theorem,
[Expression 2]

The affine transformation is established. For each segment, a plane constraint condition, that is, an affine transformation matrix A is calculated using information of segments connected by the following algorithm.
[0021]
(A) For the target segment set (SLi, SRj), the existence of the segment sets (SLi-1, SRj-1) and (SLi + 1, SRj + 1) before and after the connection is examined.
[0022]
(B) If there are segments connected to the target segment on the same straight line, they are regarded as one set of target segments (SLi ′, SRj ′) and the process returns to (a) to re-examine the connectivity.
[0023]
(C) A matrix A that minimizes ε in “Equation 3” is obtained from the corresponding points of the set of selected segments.
[Equation 3]

However, when there is no connectivity (SLi, SRj), if it is a curved segment, a plane can be defined alone, but in the case of a straight segment, parallax is expressed as a linear function of Y with a = 1, that is, Perform a linear fit.
[0024]
(D) When the error E is defined by “Equation 4”, if E is equal to or less than the threshold value, “Equation 3” is satisfied, and the process proceeds to the corresponding point correction in Step S8. Other than that, it is assumed that a plane is not formed by the combination of the segments, and returns to (c) except for the combination of segments that cause the E to increase. When only the target segment set is reached, the process proceeds to the corresponding point correction in step S8.
[Expression 4]

[0025]
(8) Corresponding point correction (S8)
Since the matrix A for satisfying the plane constraint condition is obtained in step S7, the corresponding points are corrected so as to satisfy the plane constraint condition. Specifically, XR of the corresponding point of the reference image of the target segment is converted to XR ′ using “Equation 2”.
[Equation 5]

[0026]
(9) Three-dimensional reconstruction (S9)
Since (XL, Y) of the target segment and the corrected corresponding point (XR ′, Y) are obtained, the baseline length in the standard stereo camera model is b, the focal length is f, and the three-dimensional coordinate value of the observation point is (x , Y, z), these two points are:

When this is transformed, the observation point is

Can be calculated from Since the point sequence thus obtained is projected on a plane determined by the combination of segments from the plane constraint, an error with respect to the normal direction is reduced.
[0027]
Next, processing for reducing errors on the plane is performed. If the target is a straight line segment, if it is projected on a straight line that is not parallel to the epipolar line in each image (Y ≠ constant), it is also a straight line in three dimensions, so the point projected on that plane A straight line is applied to the column, and conversion is performed so as to be a straight line in a three-dimensional space. Similarly, curve fitting is performed on a curved surface.
[0028]
FIG. 8 is a result of observing a three-dimensional shape restoration result obtained by applying the coordinate value of the corresponding point obtained in step 6 to “Equation 6” from the viewpoint of the upper left of the figure of FIG. As shown in the parallax graph in FIG. 6, rattling is seen three-dimensionally, particularly in the depth direction. On the other hand, FIG. 9 shows the result of applying the present invention observed from the same viewpoint. Since the straight-line fitting is performed in a plane-constrained manner in consideration of connectivity, the quadrangle is restored with higher accuracy without being in a torsional positional relationship with adjacent segments.
[0029]
【The invention's effect】
As described above, according to the present invention, in stereo images, parallax can be corrected from plane constraints based on the connectivity of segments, and there is an effect that a three-dimensional geometric shape can be restored with high accuracy. It can be used to give vision to systems and robots.
[Brief description of the drawings]
FIG. 1 is a process diagram showing three-dimensional reconstruction according to the present invention.
FIG. 2 is a diagram illustrating a standard stereo camera model.
FIG. 3 is a diagram illustrating an original image.
FIG. 4 is a diagram showing an image obtained by converting a standard stereo camera model system.
FIG. 5 is a diagram illustrating corresponding points.
FIG. 6 is a diagram illustrating parallax between corresponding points.
FIG. 7 is a diagram showing a set of segments before and after connecting.
FIG. 8 is a diagram illustrating a result of restoring a three-dimensional geometric shape from a parallax between corresponding points.
FIG. 9 is a diagram showing a result of applying a technique of the present invention and restoring a three-dimensional geometric shape.

Claims (5)

In a stereo image, detecting an edge and dividing it into segments by a local shape to obtain a segment feature amount;
Obtaining a correspondence relationship between stereo images by a stereo correspondence search process using a segment as a correspondence unit based on the feature amount; and
In a stereo image, a process of converting a coordinate system to make it easier to obtain a correspondence relationship;
Calculating a corresponding point from a corresponding part based on the corresponding relationship;
Based on the correspondence relationship, obtaining a plane constraint condition from the segment connection relationship;
Correcting the corresponding points in the stereo image to satisfy the plane constraint condition;
Restoring a three-dimensional shape from the relationship between corresponding points in a stereo image;
A step of performing straight / curve fitting according to segment attributes when restoring a three-dimensional shape;
A three-dimensional restoration method comprising: The step of obtaining a plane constraint condition from the connection relation of the segments comprises selecting a transformation that satisfies the plane constraint theorem in a segment connected to the target segment, and obtaining the plane constraint condition. Dimensional restoration method. 3. The method according to claim 1, wherein the step of correcting the corresponding point so as to satisfy the plane constraint condition comprises converting the corresponding point of the reference image using corresponding point conversion that satisfies the plane constraint condition. Dimensional restoration method. The step of performing straight / curve fitting according to the attribute of the segment depends on the attribute of the target segment in order to reduce the error on the plane with respect to the result after conversion to satisfy the plane constraint condition. The three-dimensional restoration method according to any one of claims 1 to 3, comprising performing straight / curve fitting processing. Means for detecting an edge in a stereo image and dividing the segment into segments by a local shape to obtain a segment feature;
Means for obtaining a correspondence between stereo images by a stereo correspondence search process using a segment as a correspondence unit based on the feature amount;
In a stereo image, a means for converting a coordinate system in order to easily obtain a correspondence relationship;
Means for calculating a corresponding point from a corresponding portion based on the corresponding relationship;
Based on the correspondence relationship, means for obtaining a plane constraint condition from the segment connection relationship;
Means for correcting corresponding points in a stereo image so as to satisfy a plane constraint condition;
Means for restoring a three-dimensional shape from the relationship of corresponding points in a stereo image;
Means for performing straight / curve fitting according to segment attributes when restoring a three-dimensional shape.

Images