JP2004030461A - Method and program for edge matching, and computer readable recording medium with the program recorded thereon, as well as method and program for stereo matching, and computer readable recording medium with the program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform accurate stereo matching. <P>SOLUTION: The surface of the ground is scanned by a TLS (three line sensor) to obtain front sight image Pf, right-under sight image Pn and a back sight image Pb of a city ground surface A. The respective images are divided into a plurality of regions R<SB>fk</SB>, R<SB>nk</SB>and R<SB>bk</SB>by texture analysis. A feature point on the region R<SB>nk</SB>is extracted to calculate a line of sight vector Vf to the feature point in the front sight image Pf. The presence/non-presence of an intersection between three-dimensional data on the city ground surface A acquired by a laser scanner and the line of sight vector Vf is investigated. After the non-presence of the intersection is confirmed, the region R<SB>nk</SB>is projected on an epitaxial polar line to perform region matching. Further, straight line edges near the regions R<SB>fk</SB>and R<SB>nk</SB>are extracted. Matching of the straight line edges is performed by narrowing down a matching range near the regions R<SB>fk</SB>and R<SB>nk</SB>. At such a time, a corresponding straight line edge is searched based upon a positional relationship between the regions R<SB>fk</SB>and R<SB>nk</SB>and the straight line edge. <P>COPYRIGHT: (C)2004,JPO

Description

Ｎ_ｎ：領域Ｒ_ｎｋが占める画素数
Ｎ_ｆ：領域Ｒ_ｆｋが占める画素数
Ｎ_ｏ：領域Ｒ_ｎｋと領域Ｒ_ｆｋとがオーバーラップする範囲の画素数
【００３２】
式（１）に表されるように、領域形状マッチング係数Ｒの分母をＮ_ｎ（領域Ｒ_ｎｋが占める画素数）でなく、Ｎ_ｎ（領域Ｒ_ｎｋが占める画素数）とＮ_ｆ（領域Ｒ_ｆｋが占める画素数）との積とすることにより、この相関係数が、領域Ｒ_ｎｋにおけるオーバーラップの程度だけを示すものではなく、領域Ｒ_ｆｋにおいてどれだけの程度でオーバーラップされているかも示すものとなる。
【００３３】
図１１は、領域形状マッチング係数Ｒの算出結果の例を示す図である。図１１において、ｘ方向にエピポーラライン上にある各領域Ｒ_ｆｋの領域番号ｋを示す。ｙ軸方向に領域Ｒ_ｎｋ中心点のエピポーラライン上における位置を示す。ｚ軸方向に領域形状マッチング係数Ｒを示す。それぞれの領域番号ｋについて領域形状マッチング係数Ｒのピークが存在するが、そのピークのうち最も高いピークを検出する。最も高いピークを有する領域Ｒ_ｆｋが領域Ｒ_ｎｋに対応すると判断し、かつこのピークが現れるエピポーラライン上の位置に領域Ｒ_ｎｋ中心点があるときに領域Ｒ_ｎｋと領域Ｒ_ｆｋがマッチングされると判断する（Ｓ５０８）。このように、領域中心点のエピポーラライン上にマッチング範囲を限定することにより、計算量を軽減させると共にマッチングの正確性を向上させることができる。また、領域形状マッチング係数Ｒを用いた領域マッチング（領域の面積及び形状に基づく領域マッチング）により、大まかな領域マッチングを少ない計算量で行うことができる。
【００３４】
Ｓ５０８で検出された最大ピークが現れるエピポーラライン上の位置の付近で領域Ｒ_ｎｋを動かしつつ、投影した領域Ｒ_ｎｋとこれと重なった領域との画素値相関係数Ｓを算出する（Ｓ５１０）。画素値相関係数Ｓは、二つの領域における濃度分布の類似性を示す相関係数であり、次の式（２）に表される。
【数２】

ｌ：領域Ｒ_ｎｋの前方視画像Ｐｆにおけるｘ方向の大きさ（ピクセル）
ｍ：領域Ｒ_ｎｋの前方視画像Ｐｆにおけるｙ方向の大きさ（ピクセル）
ｆ（ｕ，ｖ）：前方視画像Ｐｆ上の座標（ｕ，ｖ）における前方視画像Ｐｆの画
素値
ｎ（ｕ，ｖ）：前方視画像Ｐｆ上の座標（ｕ，ｖ）における領域Ｒ_ｎｋの画素値
【００３５】
画素値相関係数Ｓのピークを検出し、このときの領域Ｒ_ｎｋの位置において領域マッチングなされたものとする（Ｓ５１２）。このように、多段階の領域マッチングを行うことにより、マッチングの正確性を向上させることができる。また、画素値相関係数Ｓを用いた領域マッチング（領域の濃度分布に基づく領域マッチング）により、投影した領域とこれに対応する領域とを正確に重ね合わせることができる。
【００３６】
領域Ｒ_ｆｋ近傍の直線エッジを抽出する（Ｓ２０２）。直線エッジ抽出の具体的手順はＳ１０６と同一である。
【００３７】
Ｓ１０６で抽出された領域Ｒ_ｎｋ近傍の直線エッジを前方視画像Ｐｆに投影し、直線エッジマッチングを行う（Ｓ２０４）。ここで、直線エッジマッチングの詳細な手順を説明する。図７は、直線エッジマッチングの詳細な手順を示すフローチャートである。まず、投影された直線エッジと方向性の近い直線エッジを検索する（Ｓ７０２）。ただし、検索される範囲は、Ｓ１１２でマッチングされた領域の近傍周囲に限定される。
【００３８】
さらに、投影された直線エッジと長さの近い直線エッジを絞り込み検索する（Ｓ７０４）。
【００３９】
マッチングされた領域Ｒ_ｆｋ、領域Ｒ_ｎｋそれぞれとの位置関係に基づき対応する直線エッジを絞り込み検索する（Ｓ７０６）。
【００４０】
直線エッジ両側の平均濃度に基づき対応する直線エッジを絞り込み検索する（Ｓ７０８）。このように、Ｓ１１２でマッチングされた領域の近傍にマッチング範囲を限定することにより、すなわち領域マッチングを行った後に領域マッチングの結果を利用してエッジマッチングの範囲を限定することにより、少ない計算量で正確にマッチングすることができる。また、エッジを直線化修正した上でエッジマッチングすることにより、対応するエッジを検索するのが容易になる。加えて、多段階の直線エッジマッチングを行うことにより、マッチングの正確性を向上させることができる。
【００４１】
Ｓ１０８と同様に領域Ｒ_ｆｋ近傍の直線エッジの特徴点を抽出する（Ｓ２０６）。
【００４２】
Ｓ１０８及びＳ２０６で得られた直線エッジの特徴点のマッチングをする（Ｓ２０８）。
【００４３】
マッチングされた各特徴点の三次元座標を算出する（Ｓ２１０）。ここで、前方視画像Ｐｆと直下視画像Ｐｎにおいてマッチングされた特徴点の三次元座標算出方法を説明する。図１２は、ステレオマッチング法において対応点（特徴点）の三次元座標が算出される論理を説明する図である。理論的には前方視画像Ｐｆにおける特徴点に対する視線ベクトルと、直下視画像Ｐｎにおける特徴点に対する視線ベクトルとの交点の三次元座標を算出することにより特徴点の三次元座標を得ることができる。しかし、現実には投影中心点の測定誤差などの原因により視線ベクトルが交わらないことがある。そこで、視線ベクトル間の距離が最も近くなる点Ｏｆ、Ｏｎの中点を特徴点の三次元位置とする。
【００４４】
以下に、特徴点の三次元座標を算出する具体的手順を示す。（１）まず、前方視画像Ｐｆ上の特徴点の三次元座標と、投影中心点Ｃｆの三次元座標とを通過する直線の単位ベクトルを算出し、これから視線ベクトルＥｆ（Ｘ_ｆｐ，Ｙ_ｆｐ，Ｚ_ｆｐ）を算出する。視線ベクトルＥｆは、次の式（３）に表される。
【数３】

（Ｘ_ｆｃ，Ｙ_ｆｃ，Ｚ_ｆｃ）：投影中心点Ｃｆの三次元座標
（Ｖ_ｆｘ，Ｖ_ｆｙ，Ｖ_ｆｚ）：前方視画像Ｐｆ上の特徴点の三次元座標と、投影中心点Ｃｆの三次元座標とを通過する直線の単位ベクトル
ｔ_ｆ：パラメータ
【００４５】
（２）直下視画像Ｐｎ上の特徴点の三次元座標と、投影中心点Ｃｎの三次元座標とを通過する直線の単位ベクトルを算出し、これから視線ベクトルＥｎ（Ｘ_ｎｐ，Ｙ_ｎｐ，Ｚ_ｎｐ）を算出する。視線ベクトルＥｎは、次の式（４）に表される。
【数４】

（Ｘ_ｎｃ，Ｙ_ｎｃ，Ｚ_ｎｃ）：投影中心点Ｃｎの三次元座標
（Ｖ_ｎｘ，Ｖ_ｎｙ，Ｖ_ｎｚ）：直下視画像Ｐｎ上の特徴点の三次元座標と、投影中心点Ｃｎの三次元座標とを通過する直線の単位ベクトル
ｔ_ｎ：パラメータ
【００４６】
（３）視線ベクトル間の距離が最も近くなるときのパラメータｔ_ｆ、ｔ_ｎを算出し、これから視線ベクトル間の距離が最も近くなる点Ｏｆ、Ｏｎの三次元座標を得る。
【００４７】
（４）視線ベクトル間の距離が最も近くなる点Ｏｆ、Ｏｎの中点の三次元座標を算出する。Ｓ２１０において以上の計算を繰り返して各特徴点の三次元座標を算出する。
【００４８】
Ｓ２０８の結果から直線エッジの三次元データを取得する。このようにして三次元線分群が得られる（Ｓ２１２）。
【００４９】
領域Ｒ_ｎｋの三次元データを算出する（Ｓ２１４）。その具体的方法としては、直線エッジの特徴点と領域Ｒ_ｎｋとを紐付ける方法、あるいは領域Ｒ_ｎｋの領域全体、縁部又は角部を領域Ｒ_ｎｋの特徴点とし、ステレオマッチングによりこの特徴点の三次元座標を算出する方法などが考えられる。ここで算出する領域Ｒ_ｎｋの三次元データは大まかなものでよい。
【００５０】
すべての領域Ｒ_ｎｋについてＳ１０８からＳ２１４までの処理が完了した場合にはＳ２２２に進む。未処理の領域Ｒ_ｎｋが存在する場合には領域Ｒ_ｎｋ；ｋ＝ｘ＋１近傍の直線エッジを抽出した上で（Ｓ２１５）、Ｓ１０８に戻る（Ｓ２１６）。
【００５１】
上記の処理で得られた三次元線分群及び領域の三次元データを３Ｄワークスペース（三次元座標系）に投影する（Ｓ２１８）。
【００５２】
領域の三次元データを覆うように三次元線分群を再構成する（Ｓ２２０）。三次元線分群は上記の処理により高い精度で求めることができるが、都市の鳥瞰図を得る上では意味のないものも含まれている。しかし、領域の三次元データを参考にして三次元線分群を再構成することにより、オブジェクトとして意味のあるデータを生成することができる。
【００５３】
再構成された三次元線分群で構成される領域に建物屋上面のテンプレートを当て嵌め、建物屋上面を設定する（Ｓ２２２）。
【００５４】
得られた建物屋上面と地表面とつなぐ側壁を形成させることによりソリッド化して、建物セグメントを得る（Ｓ２２４）。
【００５５】
近接する建物セグメントをクラスタリングして、建物オブジェクトを得る（Ｓ２２６）。以上の手順を踏むことにより、都市地上面Ａの三次元データを得ることができる。
【００５６】
次に、領域マッチングの別の実施形態を説明する。図６は、領域マッチング（Ｓ１１２の処理）の別の実施形態における手順を示すフローチャートである。まず、レーザスキャナの測定結果から領域Ｒ_ｆｋ及び領域Ｒ_ｎｋの平均高さを算出する（Ｓ６０２）。上記に説明したレーザスキャナをＴＬＳに近接して設置することにより、前方視画像Ｐｆ、直下視画像Ｐｎ上のランダムポイントの高さ情報を得ることができる。
【００５７】
前方視画像Ｐｆにおける領域Ｒ_ｎｋ中心点のエピポーララインを算出する（Ｓ６０４）。
【００５８】
領域Ｒ_ｎｋの平均高さと、エピポーラライン上の領域Ｒ_ｆｋの平均高さとを比較して領域Ｒ_ｎｋに対応する領域Ｒ_ｆｋを検出する（Ｓ６０６）。
【００５９】
Ｓ６０６で検出された領域Ｒ_ｆｋと重なるように領域Ｒ_ｎｋを前方視画像Ｐｆに投影する。さらに、その周辺で領域Ｒ_ｎｋを移動させつつ、領域Ｒ_ｎｋとこれに対応する領域Ｒ_ｆｋとの画素値相関係数Ｓを算出する（Ｓ６０８）。
【００６０】
画素値相関係数Ｓのピークを検出し、このときの領域Ｒ_ｎｋとこれに対応する領域Ｒ_ｆｋとの位置関係において領域マッチングなされたものとする（Ｓ６１０）。このように、領域形状マッチング係数Ｒを用いた大まかな領域マッチングに代えて、レーザスキャナの測定結果で得られた領域の高さ情報に基づいて大まかな領域マッチングを行っても、Ｓ５０２〜５１２の領域マッチング方法と同様の効果を得ることができる。
【００６１】
なお、上記の実施形態では、上空から地上に向けてＴＬＳ走査するものであったが、建物オブジェクト群の側方から水平方向にステレオ画像の撮像を行うことによって建物オブジェット群側壁の三次元データを取得し、これから都市地上面の三次元データを取得してもよい。
【００６２】
次に、本発明の実施形態に係るエッジマッチングプログラム３及びエッジマッチングプログラム３を記録した記録媒体１３について説明する。図１３は、エッジマッチングプログラム３及びエッジマッチングプログラム３を記録した記録媒体１３の構成図である。エッジマッチングプログラム３は、その構成要素として、領域分割モジュール３２、領域マッチングモジュール３４、エッジ抽出モジュール３６及びエッジマッチングモジュール３８を含んで構成される。エッジマッチングプログラム３を後述のコンピュータに適用することにより、上記のエッジマッチング方法が実行される。
【００６３】
次に、本発明の実施形態に係るステレオマッチングプログラム４及びステレオマッチングプログラム４を記録した記録媒体１４について説明する。図１４は、ステレオマッチングプログラム４及びステレオマッチングプログラム４を記録した記録媒体１４の構成図である。ステレオマッチングプログラム４は、その構成要素として、特徴点抽出モジュール４２、視線ベクトル算出モジュール４４、隠蔽部判断モジュール４６及びステレオ画像選択モジュール４８を含んで構成される。ステレオマッチングプログラム４を後述のコンピュータに適用することにより、上記のステレオマッチング方法が実行される。
【００６４】
図１５は、エッジマッチングプログラム３又はステレオマッチングプログラム４が実行されるコンピュータ５の物理的構成を示す図である。コンピュータ５は、物理的構成要素として、中央処理装置（ＣＰＵ）５２、オペレーティングシステムを常駐させた作業用メモリ５４、読み取り装置５６、ディスプレイ５７及び入力装置５８を備える。
【００６５】
記録媒体１３又は記録媒体１４が読み取り装置５６に挿入されると、記録媒体１３又は記録媒体１４に記録された情報が読み取り装置５６からアクセス可能となり、エッジマッチングプログラム３又はステレオマッチングプログラム４が、コンピュータ５によって実行される。
【００６６】
【発明の効果】
以上説明したように、本発明により、正確なステレオマッチングを可能ならしめるエッジマッチング方法、ステレオマッチング方法等を提供することができる。
【図面の簡単な説明】
【図１】三次元鳥瞰データ取得方法Ａにより特定範囲の都市地上面Ａ（建物オブジェクト群）の三次元データを取得する手順を示す第１のフローチャートである。
【図２】三次元鳥瞰データ取得方法Ａにより特定範囲の都市地上面Ａ（建物オブジェクト群）の三次元データを取得する手順を示す第２のフローチャートである。
【図３】直線エッジ抽出の詳細な手順を示すフローチャートである。
【図４】特徴点が前方視画像Ｐｆにおける隠蔽部にあるか判断する詳細な手順を示すフローチャートである。
【図５】領域マッチングの詳細な手順を示すフローチャートである。
【図６】領域マッチングの別の実施形態における手順を示すフローチャートである。
【図７】直線エッジマッチングの詳細な手順を示すフローチャートである。
【図８】直線テンプレート及び直線テンプレートが当て嵌められる型枠を示す図である。
【図９】都市地上面の三次元データ（３Ｄレーザデータ曲面）と、視線ベクトルＶｆとが交わる様子を示す図である。
【図１０】レーザスキャナが実行される様子を示す図である。
【図１１】領域形状マッチング係数Ｒの算出結果の例を示す図である。
【図１２】ステレオマッチング法において対応点（特徴点）の三次元座標が算出される論理を説明する図である。
【図１３】エッジマッチングプログラム３及びエッジマッチングプログラム３を記録した記録媒体１３の構成図である。
【図１４】ステレオマッチングプログラム４及びステレオマッチングプログラム４を記録した記録媒体１４の構成図である。
【図１５】エッジマッチングプログラム３又はステレオマッチングプログラム４が実行されるコンピュータ５の物理的構成を示す図である。
【符号の説明】
１３、１４…記録媒体、３…エッジマッチングプログラム、３２…領域分割モジュール、３４…領域マッチングモジュール、３６…エッジ抽出モジュール、３８…エッジマッチングモジュール、４…ステレオマッチングプログラム、４２…特徴点抽出モジュール、４４…視線ベクトル算出モジュール、４６…隠蔽部判断モジュール、４８…ステレオ画像選択モジュール、５…コンピュータ、５２…ＣＰＵ、５４…作業用メモリ、５６…読み取り装置、５７…ディスプレイ、５８…入力装置。[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention records an edge matching method, an edge matching program, a computer-readable recording medium (hereinafter, simply referred to as a “recording medium”) recording the edge matching program, a stereo matching method, a stereo matching program, and a stereo matching program. It relates to a recording medium.
[0002]
[Prior art]
In the conventional stereo matching method, when matching edges in one stereo image, a corresponding edge is searched from the entire edge group detected in another stereo image.
[0003]
[Problems to be solved by the invention]
However, the conventional stereo matching method has the following problems. That is, since similar edges generally exist in an image, it is difficult to perform accurate edge matching. In addition, since the feature point in one stereo image is in the concealment part in another stereo image (the line of sight vector to the feature point when capturing another stereo image is blocked by the object), the feature point cannot be matched. There is. As a result, there is a problem that accurate stereo matching cannot be performed.
[0004]
The present invention has been made to solve the above problem, and has as its object to provide an edge matching method, a stereo matching method, and the like that enable accurate stereo matching.
[0005]
[Means for Solving the Problems]
In order to solve the above problems, the edge matching method of the present invention is used when performing a stereo matching method of acquiring three-dimensional data of an object group based on a plurality of stereo images obtained by imaging the same object group from different directions. An edge matching method for matching edges extracted from a stereo image, comprising: an area dividing step of dividing each stereo image into a plurality of areas; and matching an area in one stereo image with an area in another stereo image. The method is characterized by including a region matching step, an edge extraction step of extracting edges near the region, and an edge matching step of matching edges near each of the matched regions.
[0006]
Matching between regions having a small number of detections in an image and each having a distinctive feature amount can be performed more accurately with a smaller amount of calculation than edge matching. By narrowing down the range of edge matching using the result of such area matching, accurate edge matching can be performed. Further, by narrowing the range of edge matching, the amount of calculation can be reduced. Furthermore, the accuracy of edge matching can be further improved by performing edge matching using the positional relationship with the matched region.
[0007]
In the edge matching method of the present invention, it is preferable that in the region matching step, matching is performed based on three-dimensional data of the region acquired in advance.
[0008]
In the edge matching method of the present invention, it is preferable that in the region matching step, matching is performed based on at least one of an area and a shape of the region.
[0009]
In the edge matching method of the present invention, it is preferable that in the region matching step, matching is performed based on the density distribution of the region.
[0010]
By performing matching based on the three-dimensional data of the region or the area and shape of the region acquired by a method other than the stereo matching method, rough region matching can be performed with a small amount of calculation. By performing multi-stage matching in which this is combined with region matching based on the density of the region, the accuracy of matching can be improved.
[0011]
In the edge matching method of the present invention, the edge extracted in the edge extraction step is a straight edge, and in the edge matching step, matching is performed based on the length of the straight edge, the slope of the straight edge, and the average density on both sides of the straight edge. Preferably, this is done.
[0012]
Matching straight edges makes it easier to find corresponding edges. Further, by performing matching based on the length of the straight edge, the inclination of the straight edge, and the average density on both sides of the straight edge, edge matching can be performed more accurately.
[0013]
In order to solve the above problem, the stereo matching method of the present invention is a stereo matching method for acquiring three-dimensional data of an object group based on a plurality of stereo images obtained by imaging the same object group from different directions. A feature point extraction step of extracting feature points from a stereo image, and a gaze direction calculating a gaze vector starting from the three-dimensional coordinates of the projection center point in another stereo image and ending with the three-dimensional coordinates of the feature points acquired in advance. A vector calculation step, a line-of-sight vector, and a concealment unit determining step of determining that a feature point is in a concealment unit in another stereo image from the presence of an intersection with the three-dimensional data of the previously acquired object group, When the feature point is determined to be in the concealment part in another stereo image in the concealment part determination step, And One of the stereo image, further characterized in that to calculate the three-dimensional coordinates of the feature points by using the other stereo image.
[0014]
By investigating the presence or absence of an intersection between the line-of-sight vector for the feature point in another stereo image and the three-dimensional data of the region obtained by a method other than the stereo matching method, the feature point is mechanically hidden by the concealment unit in this stereo image. Can be determined. When the feature point is in the concealment unit, accurate edge matching is enabled by performing stereo matching of the feature point using still another stereo image.
[0015]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, with reference to the accompanying drawings, preferred embodiments of an edge matching method, an edge matching program, a recording medium on which an edge matching program is recorded, a stereo matching method, a stereo matching program, and a recording medium on which a stereo matching program is recorded according to the present invention. Will be described in detail.
[0016]
First, a three-dimensional bird's-eye view data acquisition method A to which the edge matching method and the stereo matching method of the present embodiment are applied will be described. The three-dimensional bird's-eye view data acquisition method A is for acquiring three-dimensional data of a city ground surface from a stereo image of the city ground surface captured from an aircraft flying over the city. A TLS (Three Line Sensor) is used as a stereo image pickup device. The TLS is one in which three linear CCDs are mounted on an aircraft such that they are arranged on the same flight direction line, and the three linear CCDs are respectively a forward-looking image Pf and a direct-downward-looking image of a city ground surface. Pn, a rear-view image Pb is captured. That is, by scanning the TLS, it is possible to obtain a stereo image (a front-view image Pf, a nadir-view image Pn, and a rear-view image Pb) that captures the same area of a city ground surface from different directions (see FIG. 9). ). In the three-dimensional bird's-eye view data acquisition method A, stereo matching is performed using a set of the nadir view image Pn and the forward-view image Pf or a set of the nadir view image Pn and the rear-view image Pb, thereby obtaining a three-dimensional image of the captured city ground surface. Get data. The procedure will be described below.
[0017]
1 and 2 are flowcharts showing a procedure for acquiring three-dimensional data of a city ground surface A (building object group) in a specific range by a three-dimensional bird's-eye data acquisition method A. First, a forward view image Pf, a nadir view image Pn, and a backward view image Pb of the city ground surface A are captured by TLS, and the three-dimensional coordinates of the projection center points Cf, Cn, and Cb of the linear CCD at the time of each capture are calculated. (S102). As a method for calculating the three-dimensional coordinates of the projection center points Cf, Cn, Cb, GPS or the like can be used. Note that the three-dimensional coordinates appearing below are based on the same coordinate system unless otherwise specified.
[0018]
The front-view image Pf, the nadir-view image Pn, and the back-view image Pb are each subjected to texture analysis, and then divided into a plurality of regions R _fk , R _nk , R _bk ; k = 1 to m (S104). The texture analysis is performed by detecting a portion in the image where the density (pixel value) is uniformly distributed macroscopically. Specific methods include a region expansion method and a level rising method.
[0019]
Region R _nk ; Extract straight edges near k = x (S106). Here, a detailed procedure of straight line edge extraction will be described. FIG. 3 is a flowchart showing a detailed procedure of straight line edge extraction. First, an edge in an image (a portion where a density change is sharp and where the changed portion is linearly continuous) is detected (S302). Specific methods include the Canny method and the like. Note that the straight line edge refers to a series of pixels corresponding to changed portions that are substantially aligned on a straight line and corrected for a slight deviation from the straight line.
[0020]
A straight line template is applied to the detected edge (S304). FIG. 8 is a diagram showing a straight line template and a mold to which the straight line template is fitted. The mold to which the straight line template is applied is composed of 3 pixels × 3 pixels. Assuming that pixels corresponding to the changed portions in the mold are connected in a line, the straight line template is 22.5 ° each. Eight directions λ1-8 having different angles are applied. The linear templates λ1 to λ8 are fitted for each of the molds according to the correspondence between the molds and the linear templates λ1 to λ8 shown in FIG.
[0021]
Integration of straight lines represented by the fitted straight line template (straightening correction) is performed (S306). Here, when two adjacent straight line templates are shifted by an angle difference of 22.5 °, the straight lines expressed by the straight line templates are integrated.
[0022]
Region R _nk ; Extract feature points in the straight edge group near k = x (S108). Specifically, both ends of the straight edge are set as feature points.
[0023]
It is determined whether or not each feature point is in the concealed portion in the forward-view image Pf (S110). Here, a detailed procedure for determining whether or not the feature point is located in the concealing portion in the forward-view image Pf will be described. FIG. 4 is a flowchart illustrating a detailed procedure for determining whether or not a feature point is located in the concealing unit in the forward-view image Pf.
[0024]
First, three-dimensional data (3D laser data curved surface) of a city ground surface is calculated by laser scanning (S402). FIG. 10 is a diagram showing how the laser scanner is executed. The distance between the laser scanner device mounted on the aircraft and the random point is determined by the time it takes for the aircraft flying above to emit laser light toward the random point on the ground surface and the laser light scattered at the random point to return. Then, the three-dimensional coordinates of the random point are calculated from the three-dimensional coordinates of the laser scanner device when irradiating the laser beam and the irradiation angle of the laser beam. By scanning the ground surface while repeating this process, three-dimensional data of the ground surface can be obtained. However, in general, the three-dimensional coordinates obtained here are discrete values having larger intervals than in the stereo matching method, and rough discrete three-dimensional data (3D laser data curved surface) on the ground surface is obtained by interpolating the discrete values.
[0025]
A line-of-sight vector Vf for a feature point when the front-view image Pf is captured is calculated (S404). Specifically, a vector having the three-dimensional coordinates of the projection center point Cf of the forward-view image Pf as the starting point and the three-dimensional coordinates of the feature points obtained by the laser scanner as the ending point is calculated, and this is set as the line-of-sight vector Vf.
[0026]
It is determined whether there is an intersection between the three-dimensional data (3D laser data curved surface) on the upper surface of the city and the line-of-sight vector Vf (S406). FIG. 9 is a diagram illustrating a state in which three-dimensional data (3D laser data curved surface) on the upper surface of a city intersects with a line-of-sight vector Vf. Here, the presence of the intersection means that the line of sight is blocked by another ground object when the feature point is viewed from the projection center point Cf of the forward-view image Pf. Therefore, when there is an intersection, it is determined that the feature point is located in the concealing portion of the front-view image Pf (S408). On the other hand, when there is no intersection, it is determined that the feature point is not in the concealing portion of the front-view image Pf (S409). In S110, the above processing is repeated for each feature point.
[0027]
When it is determined that none of the feature points is located in the concealing portion of the forward-view image Pf, the region R _nk is projected onto the forward-view image Pf to perform region matching (S112). On the other hand, when it is determined that any feature point is located in the concealment part of the front-view image Pf, the rear-view image Pb is selected as a stereo image instead of the front-view image Pf, and the region R _nk is set as the rear-view image Pb. Projection is performed to perform area matching (S113). The presence of occlusion in one stereo image is one of the causes that hinders accurate stereo matching. However, in the present embodiment, it is possible to mechanically detect the presence of occlusion by examining whether or not there is an intersection between the three-dimensional data (3D laser data curved surface) on the city ground surface and the line-of-sight vector Vf as described above. it can. Further, when the presence of occlusion is detected in the forward-view image Pf, the accuracy of the stereo matching is improved by selecting the rear-view image Pb as the stereo image instead of the forward-view image Pf.
[0028]
The processing of S112, 202, 204, 206, and 208 (processing when the forward-view image Pf is selected as a stereo image) will be described below, but these processings are the processing of S113, 203, 205, 207, and 209 (backward). This is the same as the process performed when the visual image Pb is selected as the stereo image), and the description of S113, 203, 205, 207, and 209 is omitted. The detailed procedure of region matching in S112 will be described. FIG. 5 is a flowchart showing a detailed procedure of the area matching.
[0029]
First, an epipolar line on the front-view image Pf of the region R _nk center point is calculated (S502). The epipolar line indicates a line of intersection between the target image and an epipolar plane (a plane including a line-of-sight vector for the target point in an image obtained by cutting out the projection center point and the target point of the target image). In S502, an epipolar line is calculated as follows. (1) The three-dimensional coordinates of the center point of the region R _nk are calculated from the two-dimensional coordinates of the center point of the region R _nk and the projection center point Cn on the nadir image Pn. (2) Three-dimensional data of a plane including the three-dimensional coordinates of the region R _nk center point, the projection center point Cn, and the projection center point Cf is calculated, and this is defined as an epipolar surface. (3) The three-dimensional data of the forward-view image Pf is calculated from the relative positional relationship between the projection center point Cf and the forward-view image Pf. (4) Calculate the three-dimensional data of the intersection line between the epipolar plane and the forward-looking image Pf, and use this as the epipolar line.
[0030]
The region R _nk is projected on the front-view image Pf such that the center point of the region R _{nk is} on the epipolar line (S504).
[0031]
While moving the center point of the region R _nk along the epipolar line, the region shape matching coefficient R between the region R _nk and each region R _fk on the epipolar line is calculated (S506). The region shape matching coefficient R is a correlation coefficient indicating the similarity of the size and shape of the two regions and the degree of overlap between the two regions, and is expressed by the following equation (1).
(Equation 1)

N _n: region _{R nk} is the number of pixels occupied by _{N f:} region _{R fk} is the number of pixels occupied by _{N o:} speed range of pixels and region _{R nk} and the region _{R fk} overlap [0032]
As represented in formula (1), instead of the denominator of the area shape matching coefficients R _{N n} (the number of pixels occupied by a region _{R nk), N n} (region _{R nk} number of pixels occupied) and _{N f} (region R _fk ), the correlation coefficient does not only indicate the degree of overlap in the region R _nk , but also how much the region R _fk overlaps. It will be shown.
[0033]
FIG. 11 is a diagram illustrating an example of a calculation result of the region shape matching coefficient R. In FIG. 11, the region number k of each region R _fk on the epipolar line in the x direction is shown. The position on the epipolar line of the center point of the region R _nk is shown in the y-axis direction. The region shape matching coefficient R is shown in the z-axis direction. A peak of the region shape matching coefficient R exists for each region number k, and the highest peak among the peaks is detected. Determines that the region R _fk with the highest peak corresponding to the region R _nk, and the region R _nk and the region R _fk are matched when the position on the epipolar line this peak appears there is a region R _nk center point A determination is made (S508). In this way, by limiting the matching range on the epipolar line at the center of the region, the amount of calculation can be reduced and the accuracy of matching can be improved. Further, by performing region matching using the region shape matching coefficient R (region matching based on the area and shape of the region), rough region matching can be performed with a small amount of calculation.
[0034]
The pixel value correlation coefficient S between the projected region R _nk and the overlapping region is calculated while moving the region R _nk near the position on the epipolar line where the maximum peak detected in S508 appears (S510). The pixel value correlation coefficient S is a correlation coefficient indicating the similarity of the density distribution in the two regions, and is represented by the following equation (2).
(Equation 2)

l: the size (pixels) of the region R _nk in the x-direction in the forward-view image Pf
m: the size (pixels) of the region R _{nk in} the forward direction image Pf in the y direction.
f (u, v): pixel value n (u, v) of forward-view image Pf at coordinates (u, v) on forward-view image Pf: region R _nk at coordinates (u, v) on forward-view image Pf Pixel value of
The peak of the pixel value correlation coefficient S is detected, and it is assumed that area matching has been performed at the position of the area R _{nk at} this time (S512). As described above, the accuracy of the matching can be improved by performing the multi-stage area matching. Further, by the area matching using the pixel value correlation coefficient S (area matching based on the density distribution of the area), the projected area and the corresponding area can be accurately overlapped.
[0036]
A straight edge near the region R _fk is extracted (S202). The specific procedure for extracting a straight edge is the same as that in S106.
[0037]
The straight line edge near the region R _nk extracted in S106 is projected on the forward-view image Pf, and straight line edge matching is performed (S204). Here, a detailed procedure of the straight line edge matching will be described. FIG. 7 is a flowchart showing a detailed procedure of straight-line edge matching. First, a straight edge having a direction similar to the projected straight edge is searched (S702). However, the search range is limited to the vicinity of the area matched in S112.
[0038]
Further, a straight line edge whose length is close to the projected straight edge is narrowed down and searched (S704).
[0039]
A corresponding straight edge is narrowed down and searched based on the positional relationship between the matched region R _fk and the region R _nk (S706).
[0040]
A corresponding straight edge is narrowed down and searched based on the average density on both sides of the straight edge (S708). As described above, by limiting the matching range to the vicinity of the region matched in S112, that is, by limiting the range of edge matching using the result of the region matching after performing the region matching, the amount of calculation is small. It can be matched exactly. In addition, by performing edge matching after straightening and correcting an edge, it is easy to search for a corresponding edge. In addition, by performing multi-step linear edge matching, the accuracy of matching can be improved.
[0041]
As in S108, feature points of a straight edge near the region _Rfk are extracted (S206).
[0042]
The feature points of the straight edges obtained in S108 and S206 are matched (S208).
[0043]
The three-dimensional coordinates of each matched feature point are calculated (S210). Here, a method of calculating three-dimensional coordinates of a feature point matched between the forward-view image Pf and the nadir-view image Pn will be described. FIG. 12 is a diagram illustrating the logic for calculating the three-dimensional coordinates of the corresponding points (feature points) in the stereo matching method. Theoretically, the three-dimensional coordinates of the feature point can be obtained by calculating the three-dimensional coordinates of the intersection of the gaze vector for the feature point in the front-view image Pf and the gaze vector for the feature point in the nadir view image Pn. However, in reality, the gaze vectors may not intersect due to a measurement error of the projection center point or the like. Therefore, the midpoint between the points Of and On where the distance between the line-of-sight vectors is the closest is defined as the three-dimensional position of the feature point.
[0044]
Hereinafter, a specific procedure for calculating the three-dimensional coordinates of the feature points will be described. (1) First, a unit vector of a straight line passing through the three-dimensional coordinates of a feature point on the front-view image Pf and the three-dimensional coordinates of the projection center point Cf is calculated, and from this, a line-of-sight vector Ef (X _fp , Y _fp , Z _fp ) is calculated. The line-of-sight vector Ef is expressed by the following equation (3).
[Equation 3]

(X _fc , Y _fc , Z _fc ): three-dimensional coordinates of the projection center point Cf (V _fx , V _fy , V _fz ): three-dimensional coordinates of a feature point on the front-view image Pf, and a cubic of the projection center point Cf A unit vector t _{f of} a straight line passing through the original coordinates: parameter
(2) A unit vector of a straight line passing through the three-dimensional coordinates of the feature points on the nadir image Pn and the three-dimensional coordinates of the projection center point Cn is calculated, and from this, the line-of-sight vectors En (X _np , Y _np , Z _np ) Is calculated. The line-of-sight vector En is represented by the following equation (4).
(Equation 4)

(X _nc , Y _nc , Z _nc ): the three-dimensional coordinates of the projection center point Cn (V _nx , V _ny , V _nz ): the three-dimensional coordinates of the feature points on the nadir image Pn, and the tertiary of the projection center point Cn Unit vector t _{n of} a straight line passing through the original coordinates: parameter
(3) The parameters t _f and t _n when the distance between the line-of-sight vectors is the closest are calculated, and the three-dimensional coordinates of the points Of and On where the distance between the line-of-sight vectors are the closest are obtained from this.
[0047]
(4) Calculate the three-dimensional coordinates of the midpoint of the points Of and On where the distance between the line-of-sight vectors is closest. In S210, the above calculation is repeated to calculate three-dimensional coordinates of each feature point.
[0048]
Three-dimensional data of a straight edge is obtained from the result of S208. Thus, a three-dimensional line segment group is obtained (S212).
[0049]
The three-dimensional data of the region R _nk is calculated (S214). As a specific method, a method applying the cord to the feature points and the region R _nk of straight edge or the entire region R _nk regions, an edge or corner to feature points in the region R _nk, the feature point by stereo matching For example, a method of calculating the three-dimensional coordinates of. The three-dimensional data of the region R _nk calculated here may be rough.
[0050]
When the processes from S108 to S214 are completed for all the regions R _nk , the process proceeds to S222. If there is an unprocessed region R _nk, a straight edge near the region R _nk ; k = x + 1 is extracted (S215), and the process returns to S108 (S216).
[0051]
The three-dimensional line group and the three-dimensional data of the region obtained by the above processing are projected onto a 3D workspace (three-dimensional coordinate system) (S218).
[0052]
A three-dimensional line group is reconstructed so as to cover the three-dimensional data of the area (S220). The three-dimensional line segment group can be obtained with high accuracy by the above-described processing, but some of them are meaningless for obtaining a bird's-eye view of the city. However, by reconstructing a three-dimensional line segment group with reference to the three-dimensional data of the region, meaningful data as an object can be generated.
[0053]
The template on the upper surface of the building is applied to the region constituted by the reconstructed three-dimensional line group, and the upper surface of the building is set (S222).
[0054]
A building segment is obtained by forming a side wall connecting the obtained building roof surface and the ground surface to solidify it (S224).
[0055]
A building object is obtained by clustering adjacent building segments (S226). By performing the above procedure, three-dimensional data of the city ground surface A can be obtained.
[0056]
Next, another embodiment of the area matching will be described. FIG. 6 is a flowchart showing a procedure in another embodiment of the area matching (processing of S112). First, the average height of the region R _fk and the region R _nk is calculated from the measurement result of the laser scanner (S602). By installing the laser scanner described above close to the TLS, it is possible to obtain height information of random points on the forward-view image Pf and the nadir-view image Pn.
[0057]
An epipolar line at the center point of the region R _nk in the forward-view image Pf is calculated (S604).
[0058]
Detecting a region _{R fk} corresponding to the region _R mean the height of the _nk, area by comparing the average height of the area _{R fk} on epipolar line _{R nk} (S606).
[0059]
The region R _nk is projected on the forward-view image Pf so as to overlap with the region R _fk detected in S606. Further, the pixel value correlation coefficient S between the region R _nk and the corresponding region R _fk is calculated while moving the region R _nk around the periphery (S608).
[0060]
It is assumed that the peak of the pixel value correlation coefficient S is detected, and the region matching is performed in the positional relationship between the region R _nk and the corresponding region R _{fk at} this time (S610). As described above, instead of performing the rough region matching using the region shape matching coefficient R, the rough region matching is performed based on the height information of the region obtained by the measurement result of the laser scanner, The same effect as in the area matching method can be obtained.
[0061]
In the above embodiment, the TLS scanning is performed from the sky to the ground. , And three-dimensional data of the city ground surface may be obtained from this.
[0062]
Next, the edge matching program 3 according to the embodiment of the present invention and the recording medium 13 on which the edge matching program 3 is recorded will be described. FIG. 13 is a configuration diagram of the edge matching program 3 and the recording medium 13 on which the edge matching program 3 is recorded. The edge matching program 3 includes, as its components, a region division module 32, a region matching module 34, an edge extraction module 36, and an edge matching module 38. The edge matching method described above is executed by applying the edge matching program 3 to a computer described below.
[0063]
Next, the stereo matching program 4 according to the embodiment of the present invention and the recording medium 14 on which the stereo matching program 4 is recorded will be described. FIG. 14 is a configuration diagram of the stereo matching program 4 and the recording medium 14 on which the stereo matching program 4 is recorded. The stereo matching program 4 includes a feature point extraction module 42, a line-of-sight vector calculation module 44, a concealment unit determination module 46, and a stereo image selection module 48 as its components. The above-described stereo matching method is executed by applying the stereo matching program 4 to a computer described below.
[0064]
FIG. 15 is a diagram illustrating a physical configuration of a computer 5 on which the edge matching program 3 or the stereo matching program 4 is executed. The computer 5 includes, as physical components, a central processing unit (CPU) 52, a working memory 54 in which an operating system is resident, a reading device 56, a display 57, and an input device 58.
[0065]
When the recording medium 13 or the recording medium 14 is inserted into the reading device 56, the information recorded on the recording medium 13 or the recording medium 14 becomes accessible from the reading device 56, and the edge matching program 3 or the stereo matching program 4 5 is performed.
[0066]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an edge matching method, a stereo matching method, and the like that enable accurate stereo matching.
[Brief description of the drawings]
FIG. 1 is a first flowchart showing a procedure for acquiring three-dimensional data of a city ground surface A (building object group) in a specific range by a three-dimensional bird's-eye data acquisition method A.
FIG. 2 is a second flowchart illustrating a procedure for acquiring three-dimensional data of a city ground surface A (building object group) in a specific range by a three-dimensional bird's-eye data acquisition method A;
FIG. 3 is a flowchart showing a detailed procedure of straight line edge extraction.
FIG. 4 is a flowchart showing a detailed procedure for determining whether or not a feature point is in a concealing portion in a forward-view image Pf.
FIG. 5 is a flowchart showing a detailed procedure of region matching.
FIG. 6 is a flowchart showing a procedure in another embodiment of region matching.
FIG. 7 is a flowchart illustrating a detailed procedure of linear edge matching.
FIG. 8 is a diagram showing a straight line template and a mold to which the straight line template is applied;
FIG. 9 is a diagram showing a state in which three-dimensional data (3D laser data curved surface) on the upper surface of a city intersects with a line-of-sight vector Vf.
FIG. 10 is a diagram showing how a laser scanner is executed.
FIG. 11 is a diagram illustrating an example of a calculation result of a region shape matching coefficient R.
FIG. 12 is a diagram illustrating logic for calculating three-dimensional coordinates of corresponding points (feature points) in the stereo matching method.
FIG. 13 is a configuration diagram of an edge matching program 3 and a recording medium 13 on which the edge matching program 3 is recorded.
FIG. 14 is a configuration diagram of a stereo matching program 4 and a recording medium 14 on which the stereo matching program 4 is recorded.
FIG. 15 is a diagram showing a physical configuration of a computer 5 on which an edge matching program 3 or a stereo matching program 4 is executed.
[Explanation of symbols]
13, 14: recording medium, 3: edge matching program, 32: area division module, 34: area matching module, 36: edge extraction module, 38: edge matching module, 4: stereo matching program, 42: feature point extraction module, 44: line-of-sight vector calculation module, 46: concealment unit determination module, 48: stereo image selection module, 5: computer, 52: CPU, 54: working memory, 56: reading device, 57: display, 58: input device.

Claims (10)

When performing a stereo matching method of acquiring three-dimensional data of the object group based on a plurality of stereo images obtained by imaging the same object group from different directions, an edge matching method that matches edges extracted from the stereo images is performed. So,
An area dividing step of dividing each of the stereo images into a plurality of areas;
An area matching step of matching the area in one stereo image with the area in another stereo image;
An edge extraction step of extracting an edge near the region;
An edge matching step of matching edges in the vicinity of each of the matched regions. In the region matching step,
The edge matching method according to claim 1, wherein the matching is performed based on three-dimensional data of the region acquired in advance. In the region matching step,
The edge matching method according to claim 1, wherein the matching is performed based on at least one of an area and a shape of the region. In the region matching step,
The edge matching method according to claim 2, wherein the matching is performed based on a density distribution of the area. The edge extracted in the edge extraction step is a straight edge,
The method according to claim 1, wherein in the edge matching step, the matching is performed based on a length of the straight edge, a slope of the straight edge, and an average density on both sides of the straight edge. Edge matching method. In a stereo matching method of acquiring three-dimensional data of the object group based on a plurality of stereo images obtained by imaging the same object group from different directions,
A feature point extraction step of extracting feature points from one stereo image,
A line-of-sight vector calculation step of calculating a line-of-sight vector having the three-dimensional coordinates of the projection center point in the other stereo image as a start point and the end point of the three-dimensional coordinates of the feature point acquired in advance,
The sight line vector, from the presence of the intersection of the previously acquired three-dimensional data of the object group, including a concealing unit determining step of determining that the feature point is in the concealing unit in the other stereo image,
When it is determined in the concealing unit determining step that the feature point is located in the concealing unit in the other stereo image, the three-dimensional feature point is determined using one of the stereo images and the other stereo image. A stereo matching method comprising calculating coordinates. An edge matching program for causing a computer to execute the edge matching method according to any one of claims 1 to 5. A computer-readable recording medium on which the edge matching program according to claim 7 is recorded. A stereo matching program for causing a computer to execute the stereo matching method according to claim 6. A computer-readable recording medium on which the stereo matching program according to claim 9 is recorded.

Images