JP2004173972A - Image processing system for strabismus endoscope and strabismus endoscopic apparatus with the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an extended sense-of-reality operation support system corresponding to a strabismus endoscope. <P>SOLUTION: The strabismus endoscopic apparatus 1 comprises the strabismus endoscope 10, an image processing device 20 and an image display device 30. The strabismus endoscope 10 comprises a camera part 11 with an imaging surface and a hard insertion part 12. The insertion part 12 is structured to be rotatable relatively to the camera part 11 around an axis L<SB>W</SB>, and an observation window 13 whose optical axis L<SB>C</SB>is inclined relatively to the axis L<SB>W</SB>is formed on the tip surface 12a of the insertion part 12. The strabismus endoscopic apparatus 1 has the image processing system for the strabismus endoscope, and the image processing system has a projection model for determining the conversion from a world coordinate system to an image coordinate system. <P>COPYRIGHT: (C)2004,JPO

Description

【００４３】
ここで、Ｒは、回転を意味する３行３列の直交行列を表し、ｔは、平行移動を意味する３次元ベクトルを表し、Ｉは単位行列を表している。
【００４４】
一方、前記Ｐ_Ｃの２次元投影点の位置ベクトルを、ｉ_Ｐ＝^ｔ（ｕ_Ｐ＝ｘ_Ｃ／ｚ_Ｃ，ｖ_Ｐ＝ｙ_Ｃ／ｚ_Ｃ，１）とすると、この投影点ｉ_Ｐから、前記画像座標系上においてこの投影点ｉ_Ｐと対応する画像点ｉ_Ｃへの対応関係は、前記内部パラメータＭを用いて、下式（９）で表される。
【００４５】
ｓｉ_Ｃ＝Ｍｉ_Ｐ……（９）
【００４６】
ここで、ｓは所定のスカラーを示している。
【００４７】
また、上式（９）における内部パラメータＭは、下式（１０）で表される。
【００４８】
【数２】

【００４９】
ここで、ｆは、焦点距離を表し、ｋ_ｕ，ｋ_ｖは、それぞれｕ軸，ｖ軸の単位長を表し、^ｔ（ｕ_０，ｖ_０）は、画像中心の座標を表している。
【００５０】
前記斜視内視鏡１０において、前記外部パラメータＴは、以下のように記述できる。まず、図２（ａ）に示すように、前記斜視内視鏡１０において、前記世界座標系Ｓ_Ｗに対して前記カメラ座標系Ｓ_Ｃが回転移動していないときの、該世界座標系からＳ_Ｗ該カメラ座標系Ｓ_Ｃへの変換をＴ_０とする。なお、図２（ａ）では、この回転の伴わないカメラ座標系をＳ_Ｃ０と表記している。
【００５１】
次いで、図２（ｂ）に示すように、世界座標系Ｓ_Ｗにおいて、前記軸線Ｌ_Ｗを回転軸として、前記撮像面に対して前記挿入部１２を角度θだけ回転移動させる場合を想定する。このときの回転を表す変換をＴ_ＲｏｔＷ（θ，ｎ_ｗ，ｃ_ｗ）とする。なお、このとき前記軸線Ｌ_Ｗは、前記２つのベクトルｎ_ｗ，ｃ_ｗによって規定される。また、図２（ｂ）では、この回転変換Ｔ_ＲｏｔＷの後、前記Ｔ_０の変換を施したカメラ座標系をＳ’_Ｃ０と表記している。
【００５２】
前記回転後のカメラ座標系Ｓ’_Ｃ０は、回転前のカメラ座標系をＳ_Ｃ０に対して相対的に回転移動している。一方、前記斜視内視鏡１０の撮像面（および前記画像座標系）の前記世界座標系Ｓ_Ｗに対する位置は変化していない。このため、カメラ座標系における回転補正が必要となる。本発明では、図３に示すように、前記カメラ座標系をＳ’_Ｃ０を、前記光軸Ｌ_Ｃを回転軸として、前記回転方向とは反対方向に角度−θだけ回転転補正する変換Ｔ_ＲｏｔＣ（−θ，ｎ_Ｃ，ｃ_Ｃ）を施すことにより、この回転補正を行う。
【００５３】
このような一連の変換を行うことにより、すなわち、上式（６）により規定された外部パラメータＴを含む、上式（５）により規定された投影モデルを適用することによって、前記世界座標系Ｓ_Ｗから前記撮像面に設定された画像座標系への変換を決定することが可能となる。
【００５４】
〈パラメータ推定方法〉
次に、前述した投影モデルで用いる各パラメータを推定する方法について説明する。まず、前述の、回転が伴わないときの変換Ｔ_０と、内部パラメータＭの推定を行う。これらの推定方法としては、従来提案されている様々な手法を適用することが可能である。本実施形態においても従来の手法、例えば、前記非特許文献１１に記載された手法を用いて、前記変換Ｔ_０と前記内部パラメータＭの推定を行う。
【００５５】
次いで、前述の、回転を伴うときの変換Ｔ_ＲｏｔＷ（θ，ｎ_ｗ，ｃ_ｗ）における、前記軸線Ｌ_Ｗの方向と通過位置をそれぞれ規定する２つのベクトルｎ_ｗ，ｃ_ｗの推定を行う。このベクトルｎ_ｗ，ｃ_ｗは、前記軸線Ｌ_Ｗを決定するパラメータなので、自由度は４となる。そこで、これらの推定にあたっては、まず、前記斜視内視鏡１０の挿入部１２に、該挿入部１２の回転に応じて前記軸線Ｌ_Ｗを中心とする円軌道上を移動する３次元位置マーカを設ける。この３次元位置マーカの円軌道の中心に対応するのが前記ベクトルｃ_ｗであり、この円軌道を含む平面の法線方向に一致し、この円軌道の中心を通るベクトルに対応するのが前記ベクトルｎ_ｗとなる。また、前記斜視内視鏡１０の前記カメラ部１１にも、該カメラ部１１の前記世界座標系Ｓ_Ｗでの位置を検出するための位置センサを設けカメラ部１１の位置を計測する。
【００５６】
次に、前記挿入部１２を回転させては、前記３次元位置マーカの位置を計測するという手順を複数回繰り返して行う。３次元位置マーカの位置計測値をｐ^ｉ _Ｗ（ｉ＝１，２，・・・ｎ）とすると、下式（１１）を満たすようなｎ_ｗを線形推定により求める。
【００５７】
【数３】

【００５８】
ここで、（，）は内積を表し、││はユークリッドノルムを表す。
【００５９】
次いで、この求められたｎ_ｗから、下式（１２）を満たすようなｃ_ｗを非線形推定、例えば、Ｌｅｖｅｎｂｅｒｇ−Ｍａｒｑｕａｄｔ法を用いて求める。
【００６０】
【数４】

【００６１】
ここで、（，）は内積を表し、││はユークリッドノルムを表す。
【００６２】
続いて、求められたｐ^ｉ _Ｗ−ｃ_ｗがなす角度から、下式（１３）により前記回転角θを求める。
【００６３】
【数５】

【００６４】
ここで、（，）は内積を表し、［，］は外積を表し、││はユークリッドノルムを表す。
【００６５】
最後に、前述の、回転補正の変換Ｔ_ＲｏｔＣ（−θ，ｎ_Ｃ，ｃ_Ｃ）における、カメラ座標系において前記光軸Ｌ_Ｃの方向を表すベクトルｎ_Ｃと、カメラ座標系において前記光軸Ｌ_Ｃの通過点の位置を表すベクトルｃ_Ｃを推定する。これらの推定にあたっては、まず、前記各ベクトルｃ_Ｃ，ｎ_Ｃ，回転角θに対応する、世界座標系上の点ｐ^ｉ _Ｗｏｂｊの計算上の画像投影点ｉ’_Ｃ（ｐ^ｉ _Ｗｏｂｊ，θ；ｎ_Ｃ，ｃ_Ｃ）を、上式（５）、（６）を用いて求める。次に、ｐ^ｉ _Ｗｏｂｊの実際の画像座標系上での位置ベクトルをｉ_Ｃ ^ｉとして、下式（１４）を満たす前記各ベクトルｃ_Ｃ，ｎ_Ｃを求める。
【００６６】
【数６】

【００６７】
〈シミュレーション実験〉
次に、前述したベクトルｃ_Ｃ，ｎ_Ｃの推定に関して行ったシミュレーション実験について説明する。シミュレーションのための生成データは、次の通りである。すなわち、観察対象として、縦横の間隔４ｍｍ、３×３に点が配列された点列の図柄をキャリブレーションに用いた。また、各点に対して標準偏差０．１ｍｍのガウスノイズを加えた。そして、斜視内視鏡の挿入部を約４５度ずつ回転させ、回転毎に４枚（４フレーム）ずつ、観察窓から約３０〜４０ｃｍ離れた位置に配置した前記図柄を撮像するという想定で、前述した投影モデルによる前記図柄の各点の投影点の図を作成した（図４）。なお、画像面（撮像面）の範囲は２００〜４００画素（ｐｉｘｅｌ）の範囲と想定した。
【００６８】
画像面上で、標準偏差０．５，１．０，１．５画素のガウスノイズをそれぞれ加えたシミュレーション結果を表１に示す。投影点の残差およびベクトルｃ_Ｃ，ｎ_Ｃの誤差は、シミュレーションで与えた真値と推定値との差である。また、与えたガウスノイズの標準偏差と、投影点の残差およびベクトルｃ_Ｃ，ｎ_Ｃの誤差との関係を明らかにするため、これらの関係を図５に示した。図５に示すように、投影点の残差およびベクトルｃ_Ｃ，ｎ_Ｃの誤差は、与えたガウスノイズの標準偏差に対して略線形であることが確かめられた。
【００６９】
【表１】

【００７０】
〈実画像における実験〉
次に、前述した投影モデルによる投影の精度を検証するために、実画像を用いて実施した複数の実験について説明する。
【００７１】
（実験１）
この実験１では、斜視角度３０度、撮像面の大きさ３４０×３４０画素の歪無し斜視内視鏡を用いた。また、観察対象は前記点列の図柄とした。斜視内視鏡の挿入部をその軸線を回転軸として約４５度ずつ回転させ、回転毎に観察対象点の画像を３枚（３フレーム）取り込み、その画像を用いてキャリブレーションを行った。そして、キャリブレーションを行った角度範囲で取得した画像の精度を調べた。観察対象点の世界座標系上での位置は、３次元センサ（Ｐｏｌａｒｉｓ；ＮｏｒｔｈｅｒｎＤｉｇｉｔａｌ Ｉｎｃ．）を用いて計測した。この３次元センサの精度は、０．３５ｍｍである。
【００７２】
この実験１で得られた結果を図６に示す。この実験１により確認された誤差は、３．０２画素であった。また、パラメータを推定する際の残差は、２．８６画素であった。この誤差は、画像面サイズの約１パーセント、１ｍｍ程度に相当する。これは、実用上の許容範囲ではあるが、前述のシミュレーション結果に比べると精度が劣る。これは、キャリブレーションに用いた画像のフレーム数が少なかったためであると考えられる。
【００７３】
（実験２）
この実験２では、前記実験１において、画像のフレーム数と各フレームにおけるサンプル点の数との関係を検証するために行った。実験に用いた器具等は、前記実験１と同じである。この実験２で得られた結果を図７に示す。図７に示すように、キャリブレーションに用いる画像のフレーム数（Ｆ）が多いほど、またフレーム毎のサンプル点数が多いほど、高い精度が得られることが確かめられた。
【００７４】
（実験３）
この実験３では、撮像面の大きさ３４０×２４０画素（０．１１ｍｍ／ｐｉｘｅｌ）の斜視内視鏡を用いた。また、観察対象は格子模様の図柄とした。斜視内視鏡の挿入部をその軸線を回転軸としてΔθずつ回転させ、回転毎に観察対象の画像を取り込み、その画像を用いてキャリブレーションを行った。また、キャリブレーションに用いた画像とは異なる５枚に画像において、画像ごとに１６点、計８０点の計測点を用いて、投影誤差（前述の投影モデルで得られた画像投影点の座標値と、観察対象の点の実際の座標を３次元センサを用いて得られた座標値との差）の平均を求めた。なお、３次元センサは、前記実験１，２で用いたものを用いた。また、用いた斜視内視鏡には、樽型歪が存在するため、前記非特許文献８に記載された方法により、歪補正を行った。
【００７５】
また、前記回転角度Δθを変化させ、回転毎に得られた画像に基づき、前述した光軸Ｌ_Ｃのキャリブレーションを行い、得られたパラメータで投影誤差を計測した。この結果を図８に示す。図８に示すように、Δθの減少と共に残差と投影誤差は減少し、Δθ＝１８度のとき誤差は４．１画素（画像面内では０．４５ｍｍ）となり、それより小さいΔθでは著しい誤差の減少は見られなかった。このとき、前記非特許文献５に記載された方法によるキャリブレーションにおける残差を求めたところ、４．２画素（画像面内では０．４６ｍｍ）であったため、投影誤差中に光軸Ｌ_Ｃのキャリブレーション誤差が占める割合は、十分小さいといえる。また、このとき得られた画像の一例を図９に示す。画像中の＋が、投影モデルにより得られた、観察対象点の画像上での位置を示している。
【００７６】
（実験５）
この実験５では、前述の投影モデルにおける軸線Ｌ_Ｗ＝（ｎ_Ｗ，ｃ_Ｗ）と、光軸Ｌ_Ｃ＝（ｎ_Ｃ，ｃ_Ｃ）の必要性を検証した。光軸Ｌ_Ｃを簡略化した投影モデル（比較例１）と、軸線Ｌ_Ｗを簡略化した投影モデル（比較例２）と、本発明の投影モデル（実施例）との比較を行った。比較にあたっては、前記実験４の手法より、Δθ＝１８度のときの投影誤差を各投影モデルにおいて求めた。その結果を表２に示す。表２に示すように、投影誤差は本発明の実施例に係る投影モデルが最も小さい。この結果から、光軸Ｌ_Ｃおよび軸線Ｌ_Ｗは、斜視内視鏡の投影モデルにおいて必要なパラメータであることがいえる。
【００７７】
【表２】

【００７８】
以上、本発明の実施形態を説明したが、本発明は上記実施形態に限定されるものではなく、種々の態様の変更が可能である。
例えば、上記実施形態では、斜視内視鏡の挿入部の回転角を検出するために３次元センサを用いているが、ロータリーエンコーダなど他の回転角検出手段を用いて回転角を検出するように構成してもよい。
【００７９】
また、上記実施形態においては、外部パラメータＴ_０、内部パラメータＭを推定する際、前記非特許文献１１に記載された手法を用いているが、この手法に代えて種々の手法、例えば、前記非特許文献１２や非特許文献「Ｉ．Ｗ． Ｆａｉｇ ：“Ｃａｌｉｂｒａｔｉｏｎ ｏｆ ｃｌｏｓｅ−ｒａｎｇｅ ｐｈｏｔｏｇｒａｍｍｅｔｒｉｃ ｓｙｓｔｅｍｓ：Ｍａｔｈｅｍａｔｉｃａｌ ｆｏｒｍｕｌａｔｉｏｎ，”Ｐｈｏｔｏｇｒａｍｍｅｔｒｉｃ Ｅｎｇ．ａｎｄ Ｒｅｍｏｔｅ Ｓｅｎｓｉｎｇ，ｖｏｌ．４１，ｐｐ．１４７９−１４８６，１９７５．」等に記載された手法を用いることができる。
【００８０】
【発明の効果】
以上、詳細に説明したように、本発明の斜視内視鏡用画像処理システムによれば、斜視内視鏡の外部空間に設定された世界座標系から斜視内視鏡の観察窓に設定されたカメラ座標系への変換を決定する外部パラメータＴを、下式によって規定したことにより、以下のような効果を奏する。
【００８１】
Ｔ＝Ｔ_０Ｔ_ＲｏｔＣ（−θ，ｎ_Ｃ，ｃ_Ｃ）Ｔ_ＲｏｔＷ（θ，ｎ_ｗ，ｃ_ｗ）
【００８２】
すなわち、この外部パラメータＴを用いることにより、挿入部の回転を伴う斜視内視鏡に対応して、世界座標系からカメラ座標系への変換を精度良く決定することが可能となる。また、この外部パラメータＴを用いることにより、斜視内視鏡に対応した投影モデルを高精度に構築することができるので、この投影モデルを用いることによって、斜視内視鏡に対応した拡張現実感手術支援システムを実現化することが可能となる。
【００８３】
また、本発明の斜視内視鏡装置によれば、本発明の斜視内視鏡用画像処理システムを備えたことにより、斜視内視鏡により得られた画像を拡張現実感手術に用いることが可能となる。
【図面の簡単な説明】
【図１】本発明の一実施形態に係る斜視内視鏡装置の構成を模式的に示す図
【図２】世界座標系からカメラ座標系への変換を模式的に示す図
【図３】カメラ座標系における回転補正を模式的に示す図
【図４】投影モデルによって得られた投影点の図
【図５】シミュレーション実験で得られた関係を示す図
【図６】実験１で得られた結果を示す図
【図７】実験２で得られた結果を示す図
【図８】実験３で得られた結果を示す図
【図９】実験４で得られた画像の一例を示す図
【符号の説明】
１ 斜視内視鏡装置
１０ 斜視内視鏡
１１ カメラ部
１２ 挿入部
１２ａ 先端面
１３ 観察窓
２０ 画像処理装置
３０ 画像表示装置
Ｌ_Ｗ 挿入部の軸線
Ｌ_Ｃ 観察窓の光軸[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an oblique endoscope image processing system for processing an image obtained by an oblique endoscope, and an oblique endoscope apparatus including the same.
[0002]
[Prior art]
With regard to endoscopic surgery having excellent low invasiveness, various augmented reality surgery support systems have been developed and disclosed (see Non-Patent Documents 1 to 6 below).
The present inventors have also developed and disclosed an augmented reality laparoscopic surgery support system using three-dimensional ultrasonic images (see Non-Patent Document 13 below).
[0003]
These surgery support systems are systems for direct-viewing endoscopes (endoscopes that look at the front, where the optical axis of the observation window coincides with the axis of the insertion section), but in endoscopic surgery, In order to secure a wide field of view, a perspective endoscope (an endoscope in which the optical axis of the observation window is inclined with respect to the axis of the insertion section and looks in an oblique direction) is often used.
[0004]
In the oblique endoscope, an optical device such as a combination of a wide-angle lens and a swing prism is used to secure a wide field of view. In order to realize an augmented reality surgery support system corresponding to such optically devised oblique mirrors, a world coordinate system (a coordinate system serving as a reference set in a three-dimensional space) must be changed to an image coordinate system. (A coordinate system set on an imaging surface such as a CCD provided in the oblique endoscope), and a conversion from the world coordinate system to the camera coordinate system required for the projection model. , An internal parameter model unique to the oblique endoscope, and a technique for estimating them are required. As for the internal parameters, it is possible to apply a model for a direct-view endoscope, and to estimate the same, it is possible to use a conventionally proposed method (for example, see Non-Patent Documents 11 and 12 below). It is. However, as for the external parameters, a model that can correspond to the oblique endoscope has not been proposed so far, and a method of estimating the model has not been proposed. Under such circumstances, conventionally, a projection model corresponding to the oblique endoscope has not been known.
[0005]
As for the estimation of the distortion parameter of the wide-angle lens, many high-precision methods have already been proposed (see Non-Patent Documents 8 to 10 below). Also, in the study of squint endoscopes, a technique has been proposed in which the field of view of each movement position is synthesized to achieve a pseudo wide angle under the condition that the movement speed of the squint endoscope is kept constant (see the non-described below). See Patent Document 7). However, it was not applicable to an augmented reality surgery support system corresponding to an oblique endoscope.
[0006]
[Non-patent document 1]
D. Dey, D .; Gobbi, P .; Slomka, K .; J. M. Surry, and T.S. M. Peters, "Automatic Fusion of Freehand Endoscopic Brain Images to Three-Dimensional Surfaces: Creating Stereoscopic Panoramas.", IEEE Trans. Med. Imag. , Vol. 21 pp. 23-30, 2002.
[0007]
[Non-patent document 2]
J. Helferty, C.I. Zhang, G .; McLennan, and W.C. Higgins, "Videoendoscopic distortion correction and it's applications to virtual guidance of endoscope", IEEE Trans. Med. Imag. , Vol. 20, pp. 605-616, 2002.
[0008]
[Non-Patent Document 3]
S. D. Buck et al. , "A system to Support laparoscopic Surgery augmented reality visualization." Springer-Verlag, 2001, Proceedings, MICCAI, pp. 691-698
[0009]
[Non-patent document 4]
R. Khadem et al. , "Endoscopy calibration and accuracy testing for 3D / 2D image registration." Springer-Verlag, 2001, Proceedings, MICCAI, pp., 1361-1365
[0010]
[Non-Patent Document 5]
P. J. Edwards, A .; P. King, C .; R. Maurer, D .; A. de Cunha, D .; J. Hawkers, D .; L. G. FIG. Hill, R.A. P. Gastion, M .; R. Fenlon, A .; J. usczyzck, A. et al. J. Strong, C.I. L. Chandler, and M.S. J. Gleeson, "Design and Evaluation of a system for microscopic-assisted guided interventions (MAGI)"
[0011]
[Non-Patent Document 6]
J. D. Stefansic, A .; J. Herline, Y .; Shyr, W.C. C. Chaman, J .; M. Fitzpatrick, B .; M. Dawant, and R.S. L. Galloway, "Registration of physical space to laparoscopic image space for use in minimally invasive hepatic surgery", IEEE Trans. Med. Imag. , Vol. 20, pp. 1012-1023, 2000.
[0012]
[Non-Patent Document 7]
H. Yamauchi et al. , “Pseudo-wide-angle endoscope visual field using“ memory endoscope ””, J. JSCAS, vol. 2, no. 2, pp. 62-68, 2000.
[0013]
[Non-Patent Document 8]
H. Haneishi, Y .; Yagihashi, and Y. Miyake “A New method for distortion correction of electronic endoscopic images”. IEEE Trans. Med. Imag. , Vol. 14, No. 3, pp. 548-555, 1995.
[0014]
[Non-Patent Document 9]
K. Vijayan et al. , "A new approach for non-linear distortion correction in endoscopy images based on least squares estimation," IEEE Trans. Med. Imag. , Vol. 18, pp. 345-354, 1995.
[0015]
[Non-Patent Document 10]
W. E. FIG. Smith et al. , "Correction of distortion in endoscopic images," IEEE Trans. Med. Imag. , Vol. 11, pp. 117-122, 1992.
[0016]
[Non-Patent Document 11]
Wang and W.S. H. Tsai, "Camera calibration by vanishing lines for 3-D computer vision", IEEE Trans. PAMI, vol. 13, no. 4, pp. 370-376, 1991.
[0017]
[Non-Patent Document 12]
R. Y. Tsai, "Aversatile camera calibration technology for high-accuracy 3-D machine vision metrology using of-the-shelf tvcameras and ans. IEEE J.I. Robot. Automat. , Vol. 3, pp. 323-344, 1987.
[0018]
[Non-patent document 13]
M. Nakamoto et al. , "3D Ultrasound System Using a Magneto-Optical Hybrid Tracker for Augmented Reality Visualization in Laparoscopic Liver Surgery". In MICCAI 2002, LNCS 2489, pp. 148-155, September 2002.
[0019]
[Problems to be solved by the invention]
As described above, the augmented reality surgery support system corresponding to the oblique endoscope has not been realized. However, in recent years, a squint endoscope is often used, particularly in the abdominal cavity, joint shaping, and otolaryngological surgery. Therefore, it is strongly desired to realize an augmented reality surgery support system corresponding to the squint endoscope. Was.
The present invention has been made in view of such circumstances, and an image processing system for a squint endoscope capable of realizing an augmented reality surgery support system corresponding to a squint endoscope, and a squint endoscope diagnosis including the same It is intended to provide a device.
[0020]
[Means for Solving the Problems]
In order to solve the above problems, an image processing system for a perspective endoscope according to the present invention includes a linearly extending insertion portion inserted into a body cavity, an observation window provided on a distal end surface of the insertion portion, and an observation target. An imaging surface on which the image of the imaging window is captured, the optical axis of the observation window is inclined with respect to the axis of the insertion portion, and the insertion portion is relative to the imaging surface with the axis as a rotation axis. An oblique endoscope configured to be rotatable in the oblique endoscope is an image processing system for processing an image obtained by the oblique endoscope,
An external parameter T for determining a conversion from the world coordinate system set in the external space of the oblique endoscope to the camera coordinate system set in the observation window is defined by the following equation (3). Things.
[0021]
T = T₀T_RotC(-Θ, n_C, C_C) T_RotW(Θ, n_w, C_w) ... (3)
[0022]
Where n_wIs a vector defining the direction of the axis in the world coordinate system, c_wIs a vector defining the passing position of the axis in the world coordinate system, n_CIs a vector defining the direction of the optical axis in the camera coordinate system, c_CIs a vector defining the passing position of the optical axis in the camera coordinate system, θ is a rotation angle when the insertion unit rotates relative to the imaging surface around the axis as a rotation axis, T₀Is a transformation representing the correspondence from the world coordinate system to the camera coordinate system when the insertion unit is not rotating relative to the imaging surface around the axis as a rotation axis;_RotWIs a transformation representing the rotation of the insertion unit when the insertion unit rotates relative to the imaging surface around the axis as the rotation axis in the world coordinate system;_RotCIndicates a transformation representing rotation correction in the camera coordinate system.
[0023]
The oblique endoscope image processing system further includes a projection model that determines a conversion from the world coordinate system to the image coordinate system set on the imaging surface, wherein the projection model is unique to the oblique endoscope. Using the internal parameter M and the external parameter T, it is possible to define the value as defined by the following equation (4).
[0024]
si_C= MTp_W…… (4)
[0025]
Where p_WIs a vector representing the position of a point on the world coordinate system, i_CIs a vector representing the position of a point on the image coordinate system, and s is a predetermined scalar.
[0026]
In addition, the oblique endoscope apparatus of the present invention includes a linearly extending insertion portion to be inserted into a body cavity, an observation window provided on a distal end surface of the insertion portion, and an imaging device for capturing an image of an observation target. And an optical axis of the observation window is inclined with respect to an axis of the insertion portion, and the insertion portion is configured to be relatively rotatable with respect to the imaging surface with the axis as a rotation axis. An oblique endoscope and an image processing system for oblique endoscope of the present invention having the above-mentioned features are provided.
[0027]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0028]
<Overview of the oblique endoscope device>
FIG. 1 is a diagram schematically illustrating a configuration of a perspective endoscope apparatus according to an embodiment of the present invention. The illustrated oblique endoscope apparatus 1 includes an oblique endoscope (oblique rigid endoscope) 10, an image processing device 20, and an image display device 30.
[0029]
The oblique endoscope 10 includes a camera unit 11 having an imaging surface (not shown) configured by an imaging device such as a CCD, and an axis L extending from the camera unit 11._WAnd a rigid insertion portion 12 extending in the direction. The insertion portion 12 is a portion to be inserted into a body cavity or the like._WIs formed obliquely. Further, the insertion portion 12 is provided with the axis L_WIs configured to be rotatable with respect to the camera unit with a rotation axis, and an observation window 13 is provided on the distal end surface 12a. The optical axis L of the observation window 13_CIs the axis L_W, The axis L_WWhen the insertion section 12 is rotated around the rotation axis, the optical axis L_CIs adapted to rotate and move relative to the imaging surface. The axis L_WIs the world coordinate system (coordinate system set in the external space of the oblique endoscope 10) S_WAnd the optical axis L_CIs the camera coordinate system (coordinate system set in the observation window 13) S_CAs defined in
[0030]
In the oblique endoscope device 1, the reflected light from the observation target is taken into the insertion section 12 through the observation window 13, and is imaged by the imaging element in the camera section 11. Then, the image signal obtained by the image sensor is subjected to signal processing in the image processing device 20, and the processed image signal is converted into an image in the image display device 30 and displayed.
[0031]
<Image processing system for oblique endoscope>
The oblique endoscope apparatus 1 includes an oblique endoscope image processing system according to an embodiment of the present invention. This image processing system for an oblique endoscope processes an image signal obtained by the oblique endoscope 10 to form an image that can be used for augmented reality surgery support. It comprises an arithmetic circuit, an arithmetic program, and the like provided therein.
[0032]
The image processing system for an oblique endoscope uses the world coordinate system S_WAnd a projection model for determining the conversion from the image data to the image coordinate system set on the imaging plane. This projection model includes an internal parameter M unique to the oblique endoscope 10 and the world coordinate system S._WFrom the camera coordinate system S_CIt is defined by the following equation (5) using the external parameter T for determining the conversion into.
[0033]
si_C= MTp_W…… (5)
[0034]
Where p_WIs a vector representing the position of a point on the world coordinate system, i_CIs a vector representing the position of a point on the image coordinate system, and s is a predetermined scalar.
[0035]
Further, in the image processing system for a perspective endoscope, the external parameter T is defined by the following equation (6).
[0036]
T = T₀T_RotC(-Θ, n_C, C_C) T_RotW(Θ, n_w, C_w) …… (6)
[0037]
Where n_wIs a vector defining the direction of the axis in the world coordinate system, c_wIs a vector defining the passing position of the axis in the world coordinate system, n_CIs a vector defining the direction of the optical axis in the camera coordinate system, c_CIs a vector defining the passing position of the optical axis in the camera coordinate system, θ is a rotation angle when the insertion unit rotates relative to the imaging surface around the axis as a rotation axis, T₀Is a transformation representing the correspondence from the world coordinate system to the camera coordinate system when the insertion unit is not rotating relative to the imaging surface around the axis as a rotation axis;_RotWIs a transformation representing the rotation of the insertion unit when the insertion unit rotates relative to the imaging surface around the axis as the rotation axis in the world coordinate system;_RotCIndicates a transformation representing rotation correction in the camera coordinate system.
[0038]
<Projection model of oblique endoscope>
Hereinafter, the projection model will be described in detail with reference to FIGS. 2 and 3 are diagrams schematically showing the concept of a projection model according to one embodiment of the present invention. FIG._WFrom the camera coordinate system S_CFIG. 3 shows the camera coordinate system S_C2 shows the rotation correction within.
[0039]
The world coordinate system S_WLet P be the position vector of the point_W=^t(X_W, Y_W, Z_W, 1), the camera coordinate system S_CLet P be the position vector of the point_CAnd the world coordinate system S_WFrom the camera coordinate system S_CLet T be the rigid affine transformation (corresponding to the external parameter T) to be transformed into_WTo P_CThe conversion to is represented by the following equation (7).
[0040]
p_C= Tp_W...... (7)
[0041]
Further, the external parameter T in the above equation (7) is represented by the following equation (8).
[0042]
(Equation 1)

[0043]
Here, R represents a 3-row, 3-column orthogonal matrix meaning rotation, t represents a three-dimensional vector meaning translation, and I represents a unit matrix.
[0044]
On the other hand, the P_CThe position vector of the two-dimensional projection point of_P=^t(U_P= X_C/ Z_C, V_P= Y_C/ Z_C, 1), this projection point i_PFrom the projection point i on the image coordinate system._PAnd the corresponding image point i_CIs represented by the following equation (9) using the internal parameter M.
[0045]
si_C= Mi_P…… (9)
[0046]
Here, s indicates a predetermined scalar.
[0047]
The internal parameter M in the above equation (9) is represented by the following equation (10).
[0048]
(Equation 2)

[0049]
Where f represents the focal length and k_u, K_vRepresents unit lengths of the u-axis and the v-axis, respectively.^t(U₀, V₀) Represents the coordinates of the center of the image.
[0050]
In the oblique endoscope 10, the external parameter T can be described as follows. First, as shown in FIG. 2A, in the oblique endoscope 10, the world coordinate system S_WWith respect to the camera coordinate system S_CFrom the world coordinate system when is not rotating_WThe camera coordinate system S_CConversion to T₀And In FIG. 2A, the camera coordinate system without rotation is represented by S_C0It is written.
[0051]
Next, as shown in FIG._WThe axis L_WIt is assumed that the insertion section 12 is rotationally moved by an angle θ with respect to the imaging surface, using as a rotation axis. The transformation representing the rotation at this time is T_RotW(Θ, n_w, C_w). At this time, the axis L_WIs the two vectors n_w, C_wDefined by Also, in FIG. 2B, this rotation conversion T_RotWAfter the T₀The camera coordinate system subjected to the transformation_C0It is written.
[0052]
The rotated camera coordinate system S '_C0Sets the camera coordinate system before rotation to S_C0Are relatively rotating with respect to. On the other hand, the world coordinate system S of the imaging surface (and the image coordinate system) of the oblique endoscope 10 is used._WThe position with respect to has not changed. Therefore, rotation correction in the camera coordinate system is required. In the present invention, as shown in FIG._C0With the optical axis L_CIs a rotation axis, the rotation T of which is corrected by the angle -θ in the direction opposite to the rotation direction._RotC(-Θ, n_C, C_C) To perform this rotation correction.
[0053]
By performing such a series of transformations, that is, by applying the projection model defined by the above equation (5) including the external parameter T defined by the above equation (6), the world coordinate system S_WTo determine the conversion to the image coordinate system set on the imaging plane.
[0054]
<Parameter estimation method>
Next, a method of estimating each parameter used in the above-described projection model will be described. First, the above-described conversion T without rotation is performed.₀And the internal parameter M is estimated. As these estimation methods, it is possible to apply various methods that have been conventionally proposed. Also in the present embodiment, the conversion T is calculated using a conventional method, for example, the method described in Non-Patent Document 11.₀And the internal parameter M is estimated.
[0055]
Next, the above-described transformation T with rotation is performed._RotW(Θ, n_w, C_w), The axis L_WVectors n defining the direction and passing position of_w, C_wIs estimated. This vector n_w, C_wIs the axis L_W, The degree of freedom is 4. Therefore, in making these estimations, first, the insertion of the axis L into the insertion portion 12 of the oblique endoscope 10 according to the rotation of the insertion portion 12 is performed._WIs provided with a three-dimensional position marker that moves on a circular orbit centered at. The vector c corresponds to the center of the circular orbit of the three-dimensional position marker._wThe vector n corresponds to the direction of the normal to the plane including the circular orbit, and corresponds to the vector passing through the center of the circular orbit._wBecomes The camera unit 11 of the oblique endoscope 10 is also provided with the world coordinate system S of the camera unit 11._WA position sensor for detecting the position at is provided to measure the position of the camera unit 11.
[0056]
Next, by rotating the insertion section 12, the procedure of measuring the position of the three-dimensional position marker is repeated a plurality of times. P is the position measurement value of the three-dimensional position markerⁱ _W(I = 1, 2,... N), n satisfying the following equation (11)_wIs obtained by linear estimation.
[0057]
(Equation 3)

[0058]
Here, (,) represents the inner product, and || represents the Euclidean norm.
[0059]
Then, this determined n_wFrom the following, c satisfying the following equation (12)_wIs determined using nonlinear estimation, for example, the Levenberg-Marquadt method.
[0060]
(Equation 4)

[0061]
Here, (,) represents the inner product, and || represents the Euclidean norm.
[0062]
Then, the required pⁱ _W-C_wThe rotation angle θ is obtained from the angle formed by the following equation (13).
[0063]
(Equation 5)

[0064]
Here, (,) represents the inner product, [,] represents the outer product, and || represents the Euclidean norm.
[0065]
Finally, the above-described rotation correction conversion T_RotC(-Θ, n_C, C_C), The optical axis L in the camera coordinate system._CVector n representing the direction of_CAnd the optical axis L in the camera coordinate system._CVector c representing the position of the passing point of_CIs estimated. In estimating these, first, each vector c_C, N_C, A point p in the world coordinate system corresponding to the rotation angle θⁱ _WobjCalculated image projection point i ′_C(Pⁱ _Wobj, Θ; n_C, C_C) Is obtained using the above equations (5) and (6). Then, pⁱ _WobjIs the position vector on the actual image coordinate system_C ⁱEach vector c satisfying the following expression (14)_C, N_CAsk for.
[0066]
(Equation 6)

[0067]
<Simulation experiment>
Next, the aforementioned vector c_C, N_CThe following describes a simulation experiment performed for estimating. The generated data for the simulation is as follows. That is, as an observation target, a pattern of a dot sequence in which dots were arranged at a vertical and horizontal interval of 4 mm and 3 × 3 was used for calibration. Gaussian noise with a standard deviation of 0.1 mm was added to each point. Then, assuming that the insertion portion of the oblique endoscope is rotated by about 45 degrees, and four symbols (4 frames) are rotated each time, and the symbol arranged at a position about 30 to 40 cm away from the observation window is imaged, A diagram of the projected points of each point of the symbol by the above-described projection model was created (FIG. 4). Note that the range of the image plane (imaging plane) was assumed to be a range of 200 to 400 pixels.
[0068]
Table 1 shows simulation results obtained by adding Gaussian noise having standard deviations of 0.5, 1.0, and 1.5 pixels on the image plane. Projected point residual and vector c_C, N_CIs the difference between the true value given by the simulation and the estimated value. Also, the standard deviation of the given Gaussian noise, the residual of the projection point and the vector c_C, N_CThese relationships are shown in FIG. 5 in order to clarify the relationship with the error. As shown in FIG. 5, the residual of the projection point and the vector c_C, N_CWas found to be approximately linear with respect to the standard deviation of the given Gaussian noise.
[0069]
[Table 1]

[0070]
<Experiment on real images>
Next, a plurality of experiments performed using actual images to verify the accuracy of projection by the above-described projection model will be described.
[0071]
(Experiment 1)
In Experiment 1, a distortion-free oblique endoscope having a perspective angle of 30 degrees and a size of an imaging surface of 340 × 340 pixels was used. The observation target was the pattern of the dot sequence. The insertion section of the oblique endoscope was rotated by about 45 degrees about its axis as a rotation axis, and three images (three frames) of the observation target point were captured every rotation, and calibration was performed using the images. Then, the accuracy of the image acquired in the angle range where the calibration was performed was examined. The position of the observation target point on the world coordinate system was measured using a three-dimensional sensor (Polaris; Northern Digital Inc.). The accuracy of this three-dimensional sensor is 0.35 mm.
[0072]
The result obtained in Experiment 1 is shown in FIG. The error confirmed in Experiment 1 was 3.02 pixels. The residual when estimating the parameters was 2.86 pixels. This error corresponds to about 1% of the image plane size and about 1 mm. This is a practically acceptable range, but the accuracy is inferior to the simulation results described above. This is probably because the number of frames of the image used for the calibration was small.
[0073]
(Experiment 2)
Experiment 2 was performed to verify the relationship between the number of image frames and the number of sample points in each frame in Experiment 1. The instruments and the like used in the experiment are the same as those in Experiment 1. The result obtained in Experiment 2 is shown in FIG. As shown in FIG. 7, it was confirmed that the higher the number of frames (F) of the image used for calibration and the larger the number of sample points for each frame, the higher the accuracy can be obtained.
[0074]
(Experiment 3)
In Experiment 3, a perspective endoscope having an imaging surface of 340 × 240 pixels (0.11 mm / pixel) was used. The observation target was a lattice pattern. The insertion portion of the oblique endoscope was rotated by Δθ about the axis thereof as a rotation axis, an image of the observation target was captured at each rotation, and calibration was performed using the image. Also, in five images different from the image used for calibration, 16 points for each image, a total of 80 measurement points, are used to calculate a projection error (coordinate values of image projection points obtained by the above-described projection model). (The difference between the actual coordinates of the observation target point and the coordinate values obtained using a three-dimensional sensor). The three-dimensional sensor used was the one used in Experiments 1 and 2. In addition, since the used oblique endoscope has barrel distortion, the distortion was corrected by the method described in Non-Patent Document 8.
[0075]
Further, by changing the rotation angle Δθ, based on the image obtained for each rotation, the aforementioned optical axis L_CWas performed, and the projection error was measured using the obtained parameters. The result is shown in FIG. As shown in FIG. 8, the residual and the projection error decrease as Δθ decreases. When Δθ = 18 degrees, the error becomes 4.1 pixels (0.45 mm in the image plane), and when Δθ is smaller, a significant error occurs. No decrease was seen. At this time, when the residual in the calibration by the method described in Non-Patent Document 5 was obtained, it was 4.2 pixels (0.46 mm in the image plane)._CCan be said to be sufficiently small. FIG. 9 shows an example of an image obtained at this time. The + in the image indicates the position on the image of the observation target point obtained by the projection model.
[0076]
(Experiment 5)
In this experiment 5, the axis L in the projection model described above was used._W= (N_W, C_W) And the optical axis L_C= (N_C, C_C) Verified the need. Optical axis L_CAnd a projection model (Comparative Example 1) obtained by simplifying_WWere compared with a projection model (Example 2) of the present invention. For comparison, the projection error when Δθ = 18 degrees was obtained for each projection model by the method of Experiment 4. Table 2 shows the results. As shown in Table 2, the projection error is smallest in the projection model according to the embodiment of the present invention. From this result, the optical axis L_CAnd axis L_WIs a necessary parameter in the projection model of the oblique endoscope.
[0077]
[Table 2]

[0078]
The embodiment of the present invention has been described above, but the present invention is not limited to the above embodiment, and various modifications can be made.
For example, in the above embodiment, the three-dimensional sensor is used to detect the rotation angle of the insertion portion of the oblique endoscope, but the rotation angle is detected by using another rotation angle detection unit such as a rotary encoder. You may comprise.
[0079]
In the above embodiment, the external parameter T₀When estimating the internal parameter M, the method described in Non-Patent Document 11 is used. Instead of this method, various methods, for example, Non-Patent Document 12 and Non-Patent Document “I.W. Faig: "Calibration of close-range photogrammetric systems: Mathematical formation," Photogrammetric Eng. And Remote Sensing, vol.
[0080]
【The invention's effect】
As described above in detail, according to the image processing system for a perspective endoscope of the present invention, the observation window of the perspective endoscope is set from the world coordinate system set in the external space of the perspective endoscope. By defining the external parameter T for determining the conversion to the camera coordinate system by the following equation, the following effects are obtained.
[0081]
T = T₀T_RotC(-Θ, n_C, C_C) T_RotW(Θ, n_w, C_w)
[0082]
That is, by using the external parameter T, it is possible to accurately determine the conversion from the world coordinate system to the camera coordinate system corresponding to the oblique endoscope with the rotation of the insertion unit. Also, by using this external parameter T, a projection model corresponding to the oblique endoscope can be constructed with high accuracy. By using this projection model, the augmented reality surgery corresponding to the oblique endoscope can be performed. It is possible to realize a support system.
[0083]
According to the oblique endoscope apparatus of the present invention, the image obtained by the oblique endoscope can be used for augmented reality surgery by providing the oblique endoscope image processing system of the present invention. It becomes.
[Brief description of the drawings]
FIG. 1 is a diagram schematically showing a configuration of a perspective endoscope apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram schematically showing conversion from a world coordinate system to a camera coordinate system.
FIG. 3 is a diagram schematically showing rotation correction in a camera coordinate system.
FIG. 4 is a diagram of projection points obtained by a projection model.
FIG. 5 is a diagram showing a relationship obtained in a simulation experiment.
FIG. 6 is a diagram showing the results obtained in Experiment 1.
FIG. 7 shows a result obtained in Experiment 2.
FIG. 8 is a diagram showing the results obtained in Experiment 3.
FIG. 9 is a diagram showing an example of an image obtained in Experiment 4.
[Explanation of symbols]
1 Oblique endoscope device
10 Oblique endoscope
11 Camera section
12 insertion section
12a Tip surface
13 Observation window
20 Image processing device
30 Image display device
L_W                Insertion axis
L_C                Optical axis of observation window

Claims (3)

An insertion portion that extends linearly to be inserted into a body cavity, an observation window provided on a distal end surface of the insertion portion, and an imaging surface on which an image of an observation target is imaged. In a perspective endoscope that is inclined with respect to the axis of the insertion portion, and the insertion portion is configured to be relatively rotatable with the axis as a rotation axis with respect to the imaging surface, An oblique endoscope image processing system for processing the obtained image,
An external parameter T for determining the conversion from the world coordinate system set in the external space of the oblique endoscope to the camera coordinate system set in the observation window is defined by the following equation (1). Image processing system for oblique endoscope.
T = T ₀ T _RotC (−θ, n _C , c _C ) T _RotW (θ, n _w , c _w ) (1)
Here, n _w is a vector that defines the direction of the axis in the world coordinate system, c _w is a vector that defines the passing position of the axis in the world coordinate system, and n _C is the vector that defines the position in the camera coordinate system. A vector that defines the direction of the optical axis, c _C is a vector that defines the passing position of the optical axis in the camera coordinate system, and θ is the relative position of the insertion section with respect to the imaging surface, with the axis as the rotation axis. The rotation angle, T ₀ , at which the insertion portion is not rotated relative to the imaging surface around the axis as the rotation axis, from the world coordinate system to the camera coordinate system. T _RotW is a conversion that represents the rotation when the insertion unit rotates relative to the imaging surface with the axis as a rotation axis in the world coordinate system, and T _RotC is: Said Shows conversion representing a rotation correction in camera coordinate system, respectively. A projection model for determining a conversion from the world coordinate system to the image coordinate system set on the imaging surface, wherein the projection model uses an internal parameter M and an external parameter T unique to the oblique endoscope. The image processing system for a perspective endoscope according to claim 1, wherein the image processing system is defined by the following equation (2).
si _C = MTp _W (2)
Here, p _W is a vector representing the position of a point on the world coordinate system, i _C is a vector representing the position of a point on the image coordinate system, s represents the predetermined scalar, respectively. An insertion portion that extends linearly to be inserted into a body cavity, an observation window provided on a distal end surface of the insertion portion, and an imaging surface on which an image of an observation target is imaged, wherein an optical axis of the observation window is 3. A perspective endoscope which is inclined with respect to an axis of the insertion section, and wherein the insertion section is configured to be relatively rotatable with respect to the imaging surface about the axis as a rotation axis. And a perspective endoscope image processing system.

Images