JP6290723B2 - Surgery support device and surgery support system - Google Patents

Description

  The present invention relates to a surgery support apparatus and a surgery support system.

  Conventionally, for example, in neurosurgery, a part of a reconstructed image such as a three-dimensional MRI (Magnetic Resonance Imaging) image is projected on a microscopic image showing an operative field and presented in a superimposed manner. Surgery support technology has been proposed.

P. J. Edwards et al, .Design and Evaluation of a system for Microscope-Assisted Guided Interventions (MAGI), MICCAI'99, LNCS 1679, pp. 842--852, 1999.

  In general, in order to superimpose an MRI image on a microscope image, registration (Registration) is required using a subject common to the microscope image and the MRI image. In the conventional surgery support technology, for example, magnification information of a microscope is used for registration. For this reason, registration is realized using a dedicated device that can output such information. Since such a dedicated device is generally expensive, if a superimposed display can be realized using an existing microscope device, a surgical support technique can be introduced into more medical sites.

  The present invention has been made in view of such problems, and an object thereof is to provide a technique for easily projecting and superimposing a three-dimensional reconstructed image on a two-dimensional microscope image.

  In order to solve the above-described problems, a surgery support apparatus according to an aspect of the present invention includes a medical image acquisition unit that acquires a reconstructed image of a region including a surgical part of a subject, and a three-dimensional surgical tool used for surgery. A three-dimensional coordinate acquisition unit that acquires typical position coordinates, and a first conversion matrix that converts coordinates of the image coordinate system set in the reconstructed image into coordinates of a coordinate system used for position coordinate measurement by the three-dimensional coordinate acquisition unit A first transformation matrix acquisition unit that acquires the image, a moving image acquisition unit that acquires a two-dimensional moving image obtained by imaging the surgical field of the subject, and a technique in the two-dimensional coordinate system set for the moving image acquired by the moving image acquisition unit An image analysis unit that acquires the two-dimensional position coordinates of the tool, a three-dimensional position coordinate of the surgical tool acquired by the three-dimensional coordinate acquisition unit, and a two-dimensional position coordinate of the surgical tool acquired by the image analysis unit Based on the above, the second variation that converts 3D position coordinates to 2D position coordinates A second transformation matrix obtaining unit that estimates a matrix, an image superimposing unit that superimposes at least a part of the reconstructed image on the two-dimensional moving image as a two-dimensional image using the first transformation matrix and the second transformation matrix; Is provided. The image analysis unit analyzes the moving image acquired by the moving image acquisition unit to detect whether at least one of the angle of view or the line-of-sight direction of the moving image has been changed. The second conversion matrix acquisition unit The second transformation matrix is estimated by detecting at least one change in the angle of view or the line-of-sight direction.

  Another aspect of the present invention is a surgery support system. The surgical operation support system includes a three-dimensional measurement device that measures three-dimensional position coordinates of a surgical instrument used for surgery, a microscope that captures a surgical field of a subject and generates a two-dimensional moving image, and information processing Device. An information processing apparatus includes a medical image acquisition unit that acquires a reconstructed image of a region including a surgical part of a subject, a three-dimensional measurement unit that acquires three-dimensional position coordinates measured by the three-dimensional measurement device, and a microscope Acquires a moving image acquisition unit that acquires a moving image generated by the image processing unit, and a first conversion matrix that converts the coordinates of the image coordinate system set in the reconstructed image into coordinates of a coordinate system used for position coordinate measurement by the three-dimensional measurement unit A first transformation matrix acquisition unit, an image analysis unit for acquiring a two-dimensional position coordinate of the surgical tool in a two-dimensional coordinate system set in a moving image generated by the microscope, and a surgical tool acquired by the three-dimensional measurement unit And a second conversion matrix for converting the three-dimensional position coordinates into the two-dimensional position coordinates based on the three-dimensional position coordinates of the tool and the two-dimensional position coordinates of the surgical instrument acquired by the image analysis unit. The second transformation matrix acquisition unit for estimating the first transformation matrix and the second transformation matrix Te, and an image superimposing unit that superimposes a two-dimensional image at least a portion of the reconstructed image on a 2D video. The image analysis unit analyzes the moving image captured by the moving image acquisition unit to detect whether at least one of the angle of view or the line-of-sight direction of the moving image acquisition unit has been changed, and the second conversion matrix acquisition unit The second transformation matrix is estimated by detecting a change in at least one of the angle of view or the line-of-sight direction.

  According to the present invention, a three-dimensional medical image can be easily projected and superimposed on a two-dimensional microscope image.

It is a figure which shows typically the whole structure of the surgery assistance system which concerns on embodiment. It is a figure which shows typically the several coordinate system used with the surgery assistance system which concerns on embodiment, and the conversion matrix between coordinate systems. It is a figure which shows typically perspective projection based on a pinhole camera model. It is a figure which shows typically the relationship between a world coordinate system and a camera coordinate system. It is a figure which shows typically the function structure of the surgery assistance apparatus which concerns on embodiment. It is a figure which shows typically the function structure of the image analysis part which concerns on embodiment. It is a flowchart explaining the flow of the surgery assistance method which concerns on embodiment. It is a flowchart explaining the flow of the moving image analysis process which concerns on embodiment.

  Surgical navigation system in the intelligent operating room enables high-precision surgery, improving the removal rate of brain tumors and improving the survival rate after 5 years. However, complete tumor removal is difficult for malignant tumors of the brain, and the survival rate after 5 years is still low. This is because malignant tumors are difficult to see with the naked eye at the boundary with the normal part of the brain, and nerves and blood vessels that should not be damaged are intertwined in a complex manner. For this reason, further engineering support is needed.

A microscope superimposed display navigation system capable of presenting intuitive position information has been reported ^{[1] [2]} . This is a system that allows an operator to intuitively present position information such as a tumor, a nerve, and a blood vessel to a surgeon by superimposing and displaying an MRI image of a patient on a camera in a microscope. However, it is necessary to perform camera calibration on the microscope camera in advance using a camera calibrator in order to perform superimposition display, and since it does not support all magnifications, superimposition display cannot be performed with high accuracy at all magnifications. In addition, there is a problem that the microscope can output the magnification one by one and that it is necessary to be a dedicated microscope with a marker in order to know the position and orientation of the microscope.

  The surgery support system according to the embodiment of the present invention realizes camera calibration in real time using surgical instrument position information obtained from a navigation system during surgery. For this reason, the surgery support system according to the present invention does not require camera calibration with a camera calibration tool in advance, and magnification and position / posture information can be obtained in real time, so that it can be used without using a dedicated microscope. .

  Hereinafter, a surgery support system according to an embodiment of the present invention will be described in more detail with reference to the accompanying drawings.

  FIG. 1 is a diagram schematically showing the overall configuration of a surgery support system 100 according to an embodiment of the present invention. The surgery support system 100 according to the embodiment includes a surgical microscope 200, a three-dimensional measurement device 300, and a surgery support device 400.

  The surgical microscope 200 enlarges and displays a moving image of a region including the surgical site of the subject 1. Hereinafter, in the present specification, “image of the region including the surgical site of the subject” is simply referred to as “surgical field”. That is, the microscope image obtained by the surgical microscope 200 is a moving image in which the surgical field is enlarged and displayed. The surgeon 2 operates with reference to the enlarged surgical field presented by the surgical microscope 200. For this reason, the surgical field viewed from the surgical microscope 200 includes the surgical instrument 3 possessed by the surgeon 2 or at least a part thereof.

  The surgical microscope 200 includes a microscope arm 4. For this reason, the surgeon 2 can change the relative position of the surgical microscope 200 with respect to the surgical site of the subject 1. In addition, the surgical microscope 200 can change and present the enlargement ratio of the surgical field.

  The surgical microscope 200 displays the surgical field on a display device such as a liquid crystal and presents it to the surgeon 2. The operation microscope 200 includes a moving image input / output terminal (not shown), and outputs a moving image of the surgical field to an external device or displays a moving image input from the outside on a display device. Can do. The surgical microscope 200 may be a general surgical microscope as long as it has such a display device and input / output terminals.

  The three-dimensional measuring apparatus 300 specifies the three-dimensional position coordinates of the subject. More specifically, the three-dimensional measuring apparatus 300 determines the position coordinates of the observation target in a three-dimensional coordinate system determined with reference to itself.

  In the surgery support system 100 according to the embodiment, the three-dimensional measuring apparatus 300 measures the position coordinates of the surgical part of the subject 1 and the position coordinates of the surgical instrument 3 held by the surgeon 2. The three-dimensional measurement apparatus 300 can be realized by using a known measurement technique such as stereo three-dimensional measurement using a binocular camera.

  Although not limited, the surgical instrument 3 imaged by the three-dimensional measurement apparatus 300 may include a marker serving as a reference for measuring the position of the surgical instrument 3. This marker is a marker used by the three-dimensional measuring apparatus 300 to measure the position of the surgical instrument 3. In this case, the three-dimensional measuring apparatus 300 detects the position of the surgical instrument 3 with reference to the measurement marker provided in the surgical instrument 3. Note that the three-dimensional measurement apparatus 300 also acquires the position coordinates of the distal end of the surgical instrument 3.

  The surgery support apparatus 400 can access a reconstructed image of a three-dimensional region including a surgical part of the subject 1 created in advance before the operation. Although not limited, the reconstructed image is, for example, an MRI image of a surgical site or a CT (Computed Tomography) image. These reconstructed images are three-dimensional image data, and are composed of so-called voxels.

  Although details will be described later, the surgical operation support apparatus 400 performs a microscopic image of the surgical microscope 200 and a three-dimensional reconstructed image in order to superimpose at least a part of the reconstructed image on the surgical field of the surgical microscope 200. Perform registration. As a result, the surgery support apparatus 400 generates a video so that the lesioned part extracted from the reconstructed image, the blood vessel of the subject 1 and the like are superimposed on the surgical field and displayed to the surgeon 2. . Thus, the surgery support apparatus 400 realizes a so-called “surgical navigation system”. The navigation system for surgery is intended to provide the surgeon 2 with positional information such as the surgical part, surgical tool 3, lesioned part, and blood vessel as reference information. Done at the discretion.

  Hereinafter, the principle of the registration process executed by the surgery support apparatus 400 according to the embodiment will be described.

FIG. 2 is a diagram schematically illustrating a plurality of coordinate systems used in the surgery support system 100 according to the embodiment and a transformation matrix between the coordinate systems. The coordinate system handled by the surgery support system 100 includes a world coordinate system C _W , a moving image coordinate system C _M , a camera coordinate system C _C , and an image coordinate system C _I.

The world coordinate system _CW is a coordinate system defined with reference to the three-dimensional measuring apparatus 300, and is a coordinate system that defines the position coordinates of an object in real space. The world coordinate system _CW may be set in any way. For example, when the three-dimensional measuring apparatus 300 is a binocular camera, a three-dimensional orthogonal coordinate system with the midpoint of the two cameras as the origin is set. . More specifically, a right hand coordinate system is set in which the viewing direction of the camera is the Z axis, the direction connecting the two cameras is the X axis, and the direction perpendicular to the Z axis and the X axis is the Y axis.

Image coordinate system C _I is a coordinate system set in the reconstructed image. As described above, the reconstructed image is three-dimensional data represented by voxels in the reconstructed image with a unique image coordinate system C _I. A plurality of feature points are set in advance in the reconstructed image used in the surgery support apparatus 400 according to the embodiment. As an example without limitation, when the surgical part is the head of the subject 1 and the reconstructed image includes the entire head of the subject 1, the position of the top of the head, the tip of the jaw, the nose and the mouth, etc. It is set as a feature point. Further, the position coordinates of the set feature points in the image coordinate system are also stored in association with the reconstructed image. These feature points may be set by a human hand or automatically using a known image recognition technique.

Camera coordinate system C _C is a coordinate system defined on the basis of the surgical microscope 200. Camera coordinate system C _C, as well as the world coordinate system C _W, a right-handed coordinate system for defining the position coordinates in the real space of an object. The moving image coordinate system _CM is a two-dimensional coordinate system set for a moving image of the surgical field generated by the surgical microscope 200.

Matrix Q is the coordinate in the image coordinate system C _I, is a matrix for converting the coordinates in the world coordinate system C _W. Hereinafter, the matrix Q will be described in more detail.

The position coordinates in the image coordinate system C _I of a feature point F, which is set on the reconstructed image and ^{(x, y, z) T} . Here, the symbol “T” represents transposition of a vector or a matrix. Further, the position coordinate in the world coordinate system _CW of the point on the real space corresponding to the feature point F is (X, Y, Z) ^T. For example, when the feature point F indicates the top of the subject 1, (x, y, z) ^T is the position coordinate of the top of the reconstructed image, and (X, Y, Z) ^T is the subject. It is a position coordinate in real space of the top of 1 head.

Hereinafter, in this specification, position coordinates are expressed using known homogeneous coordinates. That is, the position coordinates of the feature point F in the image coordinate system _{C I (x, y, z} ) and ^T, represented by four real pairs _{(x 1, x 2, x} 3, x 4) T. here,
_{x = x 1 / x 4,} y = x 2 / x 4, z = x 3 / x 4 (1)
It is. Similarly, the position coordinate of the feature point F in the world coordinate system _CW is represented by a set of four real numbers (X ₁ , X ₂ , X ₃ , X ₄ ) ^T. here,
_{X = X 1 / X 4,} Y = X 2 / X 4, Z = X 3 / X 4 (2)
It is.

と表すことができる。 The reconstructed image, since a three-dimensional data of the operative portion of the subject's 1, the position coordinates in the image coordinate system C _I of the feature point F, rotation, translation, and enlargement / reduction of by subjecting appropriate, world coordinates It can be converted into position coordinates in the system _CW . Homogeneous coordinates of the feature point F in the world coordinate system C _W, using a matrix Q of homogeneous coordinates and 4 × 4 feature points F in the image coordinate system C _I,

It can be expressed as.

で表される。Ｒは３×３の行列であり、３次元の回転行列である。Ｔは１×３のベクトルであり、並進を規定する。ｓはスケーリングパラメータであり、拡大／縮小を示すパラメータである。 Where the matrix Q is

It is represented by R is a 3 × 3 matrix and is a three-dimensional rotation matrix. T is a 1 × 3 vector and defines the translation. s is a scaling parameter and is a parameter indicating enlargement / reduction.

From equation (4), the matrix Q has 12 unknowns. Therefore, if the combination of the position coordinates of the position coordinates and the world coordinate system C _W of the image coordinate system C _I for at least four feature points are known, it is possible to estimate the matrix Q. Hereinafter, in the present specification, the matrix Q is referred to as a “first transformation matrix”.

The surgical microscope 200 projects an image of a surgical part existing in real space on a moving image 202. Referring to FIG. 3, for the matrix AP will be described for converting the coordinates of the camera coordinate system C _c to the coordinates of a moving coordinate system C _M.

FIG. 3 is a diagram schematically showing perspective projection based on a pinhole camera model. 3, the origin of the camera coordinate system C _C is set at the optical center C of the virtual pinhole camera. Z-axis of the camera coordinate system C _C image plane of the virtual pinhole camera coincides with the optical axis, the moving image 202, intersects the Z axis perpendicular at a distance f from the optical center C.

As shown in FIG. 3, the position coordinates of the camera coordinate system _{C C (X, Y, Z} ) Q point situated ^T is projected to a point q in an image plane of the moving image 202. A point q is an intersection of a line segment QC connecting the point Q and the optical center C and the image plane of the moving image 202. When the position coordinates of the moving coordinate system _{C M} of the point q and (x, ^{y) T,} the following equation (5) is satisfied.

式（５）はいわゆる透視投影を表す式である。

Expression (5) is an expression representing so-called perspective projection.

例えば、式（６）よりωｘ_１＝Ｙ_１、ωｘ_３＝Ｙ_３が得られる。これより、ｘ_１／ｘ_３＝Ｙ_１／Ｙ_３が得られる。斉次座標の定義より、左辺ｘ_１／ｘ_３＝ｘである。また、右辺はＸ／Ｚとなる。故にｘ＝Ｘ／Ｚが得られ、式（５）と一致する。同様に、式（６）よりｙ＝Ｙ／Ｚも得られる。 When the homogeneous coordinates of the point Q are (Y ₁ , Y ₂ , Y ₃ , Y ₄ ) ^T and the homogeneous coordinates of the point q are (x ₁ , x ₂ , x ₃ ) ^T , ω is an arbitrary real number, Equation (5) can be expressed by the following equation (6).

For example, ωx ₁ = Y ₁ and ωx ₃ = Y ₃ are obtained from Equation (6). Than _{this, x 1 / x 3 = Y} 1 / Y 3 is obtained. From the definition of the homogeneous coordinates, the left side x ₁ / x ₃ = x. The right side is X / Z. Therefore, x = X / Z is obtained, which is consistent with the formula (5). Similarly, y = Y / Z is also obtained from equation (6).

Incidentally, the vector (x, y) ^T represents the position coordinates of the point q on the image plane of the moving image 202, but the surgical microscope 200 actually records the video at the point q as discrete pixels. To do. Now, let the coordinates of the pixel corresponding to the point q be (u, v) ^T. At this time, the conversion from the vector (x, y) ^T to the vector (u, v) ^T includes translation for origin alignment, vertical and horizontal scale conversion, and a scale corresponding to the focal length of the lens of the surgical microscope 200. It is a combination of transformations.

Let (m ₁ , m ₂ , m ₃ ) ^T be the homogeneous coordinates of the vector (u, v) ^T. That is, u = m ₁ / m ₃ and v = m ₂ / m ₃ . If the coordinates of the image center of the moving image 202 are (u ₀ , v ₀ ), the focal lengths are f, u, and the scale factors in the v direction are k _u , k _v , respectively, (m ₁ , m ₂ , m ₃ ) The relationship between ^T and (x ₁ , x ₂ , x ₃ ) ^T can be expressed by the following equation (7).

ここで、

は、内部パラメータ行列と呼ばれる。また、

は、透視投影を表す行列である。以上より、行列ＡＰは、カメラ座標系Ｃ_Ｃの斉次座標を、動画像２０２の画像面を構成する画素座標に変換する行列である。 From the equations (6) and (7), the following equation (8) is obtained.

here,

Is called the internal parameter matrix. Also,

Is a matrix representing perspective projection. Thus, the matrix AP is the homogeneous coordinates of the camera coordinate system C _C, a matrix for transforming the pixel coordinates constituting the image plane of the moving image 202.

  Next, the matrix M will be described with reference to FIG.

Figure 4 is a diagram schematically showing the relationship between the world coordinate system C _W and the camera coordinate system C _C. As described above, the surgical microscope 200 includes the microscope arm 4. For this reason, the relative positional relationship of the surgical microscope 200 with respect to the three-dimensional measurement apparatus 300 can change.

4, the coordinate transformation from the world coordinate system C _W to the first camera coordinate system C _C is converted by the three-dimensional rotation R and 3-dimensional translation T. Similarly, the coordinate transformation from the world coordinate system _CW to the second camera coordinate system C _C ′ is transformed by a three-dimensional rotation R ′ and a three-dimensional translation T ′.

ここでＭは３次元の回転行列Ｒと、並進を規定するベクトルＴを用いて、以下の式（１０）で表される。

Now, the homogeneous coordinates of the point Q in the camera coordinate system _{C C (Y 1, Y 2} , Y 3, Y 4) T, the homogeneous coordinates of the point Q in the world coordinate system _{C W (X 1, X 2} , X ₃ , X ₄ ) ^T. In this case, coordinate transformation from the world coordinate system C _W to a camera coordinate system C _C can be expressed by using the matrix M of 4 × 4, the following equation (9).

Here, M is expressed by the following equation (10) using a three-dimensional rotation matrix R and a vector T that defines translation.

From the equations (8), (9), and (10), the homogeneous coordinates (X ₁ , X ₂ , X ₃ , X ₄ ) ^T in the world coordinate system C _W are used to construct the image plane of the moving image 202. The following equation (11) is converted into the homogeneous coordinates (m ₁ , m ₂ , m ₃ ) ^T of the pixel to be converted.

In equation (11), the matrix P _P = APM is a 3 × 4 matrix. Therefore, the elements of the matrix _{P P} is the following equation (12).

Substituting equation (12) into equation (11) yields the following equations (13), (14), and (15).
ωm ₁ = p ₁₁ X ₁ + p ₁₂ X ₂ + P ₁₃ X ₃ + P ₁₄ X ₄ (13)
ωm ₂ = p ₂₁ X ₁ + p ₂₂ X ₂ + P ₂₃ X ₃ + P ₂₄ X ₄ (14)
ωm ₃ = p ₃₁ X ₁ + p ₃₂ X ₂ + P ₃₃ X ₃ + P ₃₄ X ₄ (15)

同様に、式（１４）、式（１５）、および斉次座標の定義より、以下の式（１７）を得る。

From the formulas (13), (15), and the definition of homogeneous coordinates, the following formula (16) is obtained.

Similarly, the following formula (17) is obtained from the formula (14), the formula (15), and the definition of the homogeneous coordinates.

ここで、ベクトルｐは行列Ｐ_Ｐの要素から構成されるベクトルであり、以下の式（１９）で表される１２×１のベクトルである。

From the equations (16) and (17), the following equation (18) is obtained.

Here, the vector p is a vector composed of elements of the matrix P _P, a vector of 12 × 1 represented by the following formula (19).

In the equation (18), the components of the matrix D are the homogeneous coordinates in the world coordinate system _CW and the coordinates of the corresponding pixels, and can be observed. Now, when we consider a certain 12 × 1 vector p,
Dp = e (20)
Suppose that Here, e is a 2 × 1 vector that becomes (e ₁ , e ₂ ) ^T. In the formula (20), if Motomare the vector p 2 norm of e is minimized, the minimum square error solution of 2-norm of the error is the optimal solution of the matrix P _P in terms of minimizing is obtained. When this least square error solution is P _opt , P _opt is expressed by the following equation (21).

ここで、λはラグランジュの未定乗数である。 Since equation (21) has a trivial solution of p = 0, the following equation (22) is obtained by using the Lagrange's undetermined multiplier method with the constraint that the 2-norm of p has a positive value.

Here, λ is a Lagrange multiplier.

^{When p} T ^D T Dp + ^λ in the equation (21) to ^(1-p T p) was placed and C, p C is an extreme value it is a solution to obtain. When C is differentiated with respect to p, the following equation (23) is obtained.

Equation (23) holds when the vector p is an eigenvector of the matrix D ^T D. Thus, by obtaining an eigenvector of the matrix D ^{T D,} to obtain an estimate of the matrix P _P of converting the homogeneous coordinates in the world coordinate system C _W in homogeneous coordinates of pixels constituting the image plane of the moving image 202 it can. Herein below, the matrix P _P is referred to as "second conversion matrix". Various solvers for obtaining eigenvectors of the matrix exist and can be used. By using these solvers, the eigenvectors of the matrix D ^T D can be calculated within a sufficiently short time compared to the time required for the operation. Therefore, even if at least one of the angle of view or line of sight direction of the surgical microscope 200 (imaging direction) is changed, it is possible to re-calculate an estimate of the matrix P _P in real time during surgery.

Since the equation (20) shows the relationship between the homogeneous coordinates of one point in the world coordinate system _CW and the coordinates of the pixel on the moving image corresponding to the point, the size of the matrix D is 2 × 12. It has become. When the homogeneous coordinates in the world coordinate system _{CW and} the coordinates of the corresponding pixels on the moving image are known for a plurality of points, the matrix D can be constructed using these coordinates. For example, when the homogeneous coordinates in the world coordinate system _{CW and} the coordinates of the corresponding pixels on the moving image are known for N points (N is a natural number), the size of the matrix D is 2N × 12. The more the number of the corresponding points, are more likely to obtain an estimate of the precise matrix P _P.

As described above, the components of the matrix D are the homogeneous coordinates in the world coordinate system _CW and the coordinates of the corresponding pixels. For example, when the surgeon 2 moves the surgical microscope 200 or changes the magnification of the lens, the internal parameter matrix A is changed. As a result, the second transformation matrix _PP is also changed. In such a case, for example, the position coordinates of the surgical instrument 3 in the world coordinate system C _W, obtains the position coordinates in the image coordinate system C _M corresponding thereto to first configure the matrix D in equation (23). Then, by obtaining the eigenvector of the matrix D ^{T D,} it is possible even during operation again estimating a second transformation matrix P _P in real time.

From equation (3) and equation (11), homogeneous coordinates _x 1 in the image coordinate system _{C I} set in the reconstructed _image, x _2, x _3, the x ^{4) T,} constitutes the image plane of the moving image 202 The equation for converting to the homogeneous coordinates (m ₁ , m ₂ , m ₃ ) ^T of the pixel to be expressed is expressed by the following equation (24).

Equation (24), a first transformation matrix Q, by using a second transformation matrix P _P, be associated to the moving image a reconstructed image surgical microscope 200 produces, i.e. moving image reconstructed image It shows that it can be registered. Therefore, by using Expression (24), it is possible to superimpose a part of the reconstructed image (for example, a lesion, blood vessel, nerve, etc.) on the surgical field indicated by the surgical microscope 200. A part of the reconstructed image to be superimposed on the surgical field indicated by the surgical microscope 200 may be specified manually or using an image recognition technique in advance.

  The principle of the registration process executed by the surgery support apparatus 400 according to the embodiment has been described above. Next, the functional configuration and operation of the surgery support apparatus 400 according to the embodiment will be described.

  FIG. 5 is a diagram schematically illustrating a functional configuration of the surgery support apparatus 400 according to the embodiment. The surgery support apparatus 400 includes a medical image acquisition unit 402, a first conversion matrix acquisition unit 404, a three-dimensional coordinate acquisition unit 406, an image superimposition unit 408, a moving image acquisition unit 410, an image analysis unit 412, and a second conversion matrix acquisition unit 414. Is provided. As a non-limiting example, the surgery support apparatus 400 can be realized by using a tablet, a smartphone, or the like in addition to a PC (Personal Computer), a computer constituting a PACS (Picture Archiving and Communication System), and a workstation.

  FIG. 5 shows a functional configuration for realizing the surgery support apparatus 400 according to the embodiment, and other configurations are omitted. In FIG. 5, each element described as a functional block for performing various processes is configured by a CPU (Central Processing Unit), a main memory, and other LSI (Large Scale Integration) of the surgery support apparatus 400 in terms of hardware. can do. In terms of software, it is realized by a program loaded in the main memory. Accordingly, it is understood by those skilled in the art that these functional blocks can be realized in various forms, and is not limited to any one.

  The medical image acquisition unit 402 acquires a three-dimensional reconstructed image of an area including the surgical part of the subject. Although not shown, the medical image acquisition unit 402 is connected to a network such as a facility where the surgery support apparatus 400 is installed, and acquires a reconstructed image via the network.

The three-dimensional coordinate acquisition unit 406 acquires the three-dimensional position coordinates of the surgical instrument 3 used for the operation from the three-dimensional measurement apparatus 300. More specifically, the three-dimensional coordinate acquisition unit 406 is connected to the three-dimensional measurement apparatus 300 by wire, and acquires the position coordinates of the surgical instrument 3 by wire. The coordinates acquired by the three-dimensional coordinate acquisition unit 406 are the position coordinates of the surgical instrument 3 in the world coordinate system _CW . When the 3D coordinate acquisition unit 406 and the 3D measurement device 300 are connected via a wired or wireless network, the position coordinates of the surgical instrument 3 may be acquired via the network.

First transformation matrix acquisition unit 404 for the coordinates of the image coordinate system C _M set in the reconstructed image 3-dimensional coordinate acquiring unit 406 converts the coordinates of the world coordinate system C _W used in the position coordinate measuring first A transformation matrix Q is obtained. The first transformation matrix Q is estimated in advance using the above equations (3) and (4) and stored in a storage unit (not shown) in the surgery support apparatus 400. The first transformation matrix acquisition unit 404 refers to the storage unit in the surgery support apparatus 400 and acquires the first transformation matrix Q. Alternatively, the first transformation matrix acquisition unit 404 may estimate and acquire the first transformation matrix Q using the position coordinates of the feature points set in advance in the reconstructed image.

The moving image acquisition unit 410 acquires a two-dimensional moving image 202 obtained by photographing the surgical field of the subject 1. Here, the moving image acquired by the moving image acquisition unit 410 is a moving image generated and output by the surgical microscope 200. The image analysis unit 412, the moving coordinate system C _M is a two-dimensional coordinate system set in the moving image 202 moving image obtaining unit 410 obtains, acquires two-dimensional position coordinates of the surgical instrument 3. Details of the image analysis unit 412 will be described later.

The second transformation matrix acquisition unit 414 acquires estimated 2 transformation matrix _{P P.} More specifically, the second transformation matrix acquisition unit 414 performs the three-dimensional position coordinates of the surgical tool 3 acquired by the three-dimensional coordinate acquisition unit 406 and the two-dimensional position of the surgical tool 3 acquired by the image analysis unit 412. The second transformation matrix _PP is estimated by constructing the matrix D in Expression (18) based on the correct position coordinates and obtaining the vector p satisfying Expression (23).

Image superimposing unit 408, by using a first transformation matrix Q and a second transformation matrix P _P, according to equation (24), as a two-dimensional image at least a portion of the reconstructed image on the moving image 202 of the surgical field Superimpose. The image superimposing unit 408 outputs the moving image 202 on which a part of the reconstructed image is superimposed to the surgical microscope 200. Thereby, the surgeon 2 who observes the surgical site of the subject 1 using the surgical microscope 200 can observe an image in which a part of the reconstructed image is superimposed on the surgical field.

Here, for example, the surgeon 2 changes the relative position of the surgical microscope 200 with respect to the surgical site of the subject 1 or changes the magnification of the lens of the surgical microscope 200 (that is, changes the focal length). Suppose that Since the second transformation matrix P _P is also changed along with the change, the second transformation matrix acquisition unit 414 is preferably re-estimating the second transformation matrix P _P.

Therefore, the image analysis unit 412 analyzes the moving image 202 acquired by the moving image acquisition unit 410 from the surgical microscope 200, and detects whether at least one of the angle of view or the line-of-sight direction of the moving image 202 has been changed. The second transformation matrix acquisition unit 414, triggered by the image analysis unit 412 detects at least one of the change in the angle of view or line of sight direction, estimating a second transformation matrix P _P. When the relative position of the surgical microscope 200 is changed or the magnification of the lens of the surgical microscope 200 is changed, the angle of view or the line-of-sight direction of the moving image 202 also changes. Thus, by the image analysis unit 412 and the second transformation matrix acquisition unit 414 in response to detecting at least one of changing the angle of view or line of sight direction of the moving image 202 to estimate the second transformation matrix P _P, reconstruction Appropriate registration between the image and the moving image 202 can be maintained.

  FIG. 6 is a diagram schematically illustrating a functional configuration of the image analysis unit 412 according to the embodiment. The image analysis unit 412 includes a surgical instrument specifying unit 420, a background region specifying unit 422, a difference image acquiring unit 424, and a difference image analyzing unit 426.

The surgical instrument specifying unit 420 detects the surgical instrument 3 shown in the moving image 202 acquired from the surgical microscope 200. More specifically, the surgical instrument specifying unit 420 analyzes the moving image 202 acquired from the surgical microscope 200 to acquire the position coordinates of the tip of the surgical instrument 3 as the coordinates of the pixels constituting the moving image 202. . This can be realized by the following procedure using image processing such as known Hough transform ^[3] and SURF ^[4] .

  First, the surgical instrument specifying unit 420 recognizes the surgical instrument 3 by applying a Hough transform used for straight line detection to the acquired moving image 202, and based on the Hough transform, a surgical instrument is obtained from ROI (Region Of Interest). Specify the range near 3. Subsequently, the surgical instrument specifying unit 420 extracts the feature point of the surgical instrument 3 by applying the SURF to the ROI range, and detects the minimum horizontal point of the extracted feature point as the distal end position of the surgical instrument 3. The surgical instrument specifying unit 420 detects the tip position of the surgical instrument 3 for each frame of the acquired moving image 202 or for each predetermined frame, and outputs the detection result to the second transformation matrix acquisition unit 414.

  The background area specifying unit 422 specifies a background area for each frame of the moving image 202. Here, the “background region” is a region in which a part other than the surgical instrument 3 is shown in the moving image 202. In general, even if the position and magnification of the surgical microscope 200 are fixed, the position of the surgical instrument 3 changes during the operation. However, when the position and magnification of the surgical microscope 200 are fixed, the background region is considered to change little. Therefore, whether or not the background region changes can be an indicator of whether or not the position and magnification of the surgical microscope 200 have been changed.

  Therefore, the difference image acquisition unit 424 acquires a difference image of the background image between different frames specified by the background region specifying unit 422. The difference image analysis unit 426 acquires the difference image acquisition unit 424 and analyzes the difference image to detect whether at least one of the angle of view or the line-of-sight direction of the moving image 202 has been changed. As a specific example, the difference image analysis unit 426 generates the square of the pixel values of the difference image or the sum of absolute values for each pair of frames. The difference image analysis unit 426 monitors the generated sum and estimates that at least one of the angle of view or the line-of-sight direction of the moving image has been changed when the sum changes by a predetermined amount.

  Here, the “predetermined amount” is a calculation method of the sum, and a reference threshold used by the difference image analysis unit 426 to estimate a change in the angle of view or the line-of-sight direction of the moving image 202. The specific value of the predetermined amount may be determined by experiment in consideration of the number of pixels of the moving image 202, the shape and size of the surgical instrument 3, the type of operation assumed, and the like.

  FIG. 7 is a flowchart for explaining the flow of the surgery support method according to the embodiment. The process in this flowchart starts when the surgery support apparatus 400 is activated, for example.

The medical image acquisition unit 402 acquires a three-dimensional reconstructed image relating to a region including the surgical part of the subject 1 (S2). First transformation matrix acquisition unit 404 acquires the first transformation matrix Q to transform the coordinates of the image coordinate system C _M to the coordinates of the world coordinate system C _W (S4).

  The three-dimensional coordinate acquisition unit 406 acquires the three-dimensional position coordinates of the surgical instrument 3 used for surgery (S6). The moving image acquisition unit 410 acquires a two-dimensional moving image 202 obtained by photographing the surgical field of the subject 1 (S8).

The surgical instrument specifying unit 420 in the image analysis unit 412 analyzes the moving image 202 acquired from the surgical microscope 200, and acquires the position coordinates of the distal end of the surgical instrument 3 as two-dimensional position coordinates (S10). The second transformation matrix acquisition unit 414, the position coordinates in the world coordinate system C _W of the surgical instrument 3, the position coordinates of the moving coordinate system C _M of the surgical instrument 3 on the basis of three-dimensional in the world coordinate system C _W the Do coordinates, obtains a second transformation matrix _{P P} of converting the position coordinates in the moving coordinate system _{C M} (S12).

Image superimposing unit 408, based on a first transformation matrix Q and a second transformation matrix _{P P,} superimposes the reconstructed image to the moving image 202 to be displayed on the surgical microscope 200 (S14). The image analysis unit 412 analyzes the moving image 202 acquired from the surgical microscope 200 in order to detect whether at least one of the angle of view or the line-of-sight direction of the surgical microscope 200 has been changed (S16).

  When the image analysis unit 412 detects that at least one of the angle of view or the line-of-sight direction of the moving image 202 has been changed (Y in S18), the process returns from step S16 to the process from step 6 to step 16 is executed. Is done. When the angle of view and the line-of-sight direction of the moving image 202 are not changed (N in S18), the process from Step S14 to Step S18 is repeated while the operation is not completed (N in S20).

  When the surgery is finished (Y in S20), the processing in this flowchart is finished.

  FIG. 8 is a flowchart for explaining the flow of the moving image analysis process according to the embodiment, and is a view for explaining the flow of the process of step S16 in FIG. 7 in more detail.

  The background area specifying unit 422 specifies a background area for each frame of the acquired moving image 202 (S22). The difference image acquisition unit 424 obtains a difference image of the background region using two different frames as a pair (S24).

  The difference image analysis unit 426 analyzes the difference image acquired by the difference image acquisition unit 424 and detects whether or not at least one of the angle of view or the line-of-sight direction of the surgical microscope 200 has been changed (S28). When the difference image analysis unit 426 detects whether or not at least one of the angle of view or the line-of-sight direction has been changed, the processing in this flowchart ends.

  As described above, according to the surgery support apparatus 400 according to the embodiment, it is possible to provide a technique for easily projecting and superimposing a three-dimensional reconstructed image on a two-dimensional microscope image.

  In particular, the surgery support apparatus 400 according to the embodiment can superimpose the reconstructed image on the microscope image without using a microscope specially prepared for superimposed display. In addition, in the superimposed display, for example, registration of the reconstructed image and the microscope image can be realized without using information such as the zoom information of the microscope. Since this registration is completed in a short time, highly accurate registration can be maintained even if at least one of the angle of view and the line-of-sight direction of the microscope is changed during the operation.

  As mentioned above, although this invention was demonstrated based on embodiment, embodiment only shows the principle and application of this invention. In the embodiment, many modifications and arrangements can be made without departing from the spirit of the present invention defined in the claims.

  In the above description, a case has been described in which a three-dimensional reconstructed image such as an MRI image or a CT image is registered with a microscope image, and at least a part of the reconstructed image is superimposed on the microscope image. However, the principle of the registration process described above can be applied to cases other than the case where the reconstructed image is superimposed on the microscope image. For example, in the medical range, instead of the surgical microscope 200, for example, a reconstructed image may be superimposed on an endoscope image.

  The principle of the registration process described above is not limited to medical use, and can be widely applied when three-dimensional data is superimposed on a two-dimensional image. For example, it can be used when superimposing an AR (Augmented Reality) image on the real world, such as a glasses-type wearable computer.

[References]
[1] Inoue et al., Future Prospects and Challenges of Microscopic Superposition Navigation in Neurosurgery-Comparison with Contour Superposition Display of Commercial Systems-, Journal of Japanese Society of Computer Aided Surgery, Vol.13 (3), pp. 211-212 , 2011.
[2] PJ Edwards et al., Design and Evaluation of a System for Microscope-Assisted Guided Interventions (MAGI), MICCAI '99, LNCS 1679, pp. 842-852, 1999.
[3] Kenji Murakami, Acceleration of line extraction by Hough transform using inter-image arithmetic, IEEJ Transactions 133 (8), 1539-1548, 2013.
[4] Hiroyoshi Fujiyoshi, Gradient-based feature extraction: SIFT and HOG, Information Processing Society of Japan CVIM (160): 211-224, 2007.

  3 surgical tools, 100 surgery support system, 200 surgical microscope, 300 3D measurement device, 400 surgery support device, 402 medical image acquisition unit, 404 first transformation matrix acquisition unit, 406 3D coordinate acquisition unit, 408 image superposition unit 410 moving image acquisition unit, 412 image analysis unit, 414 second transformation matrix acquisition unit, 420 surgical tool identification unit, 422 background region identification unit, 424 difference image acquisition unit, 426 difference image analysis unit.

Claims (5)

A medical image acquisition unit for acquiring a reconstructed image of a region including the surgical part of the subject;
A three-dimensional coordinate acquisition unit for acquiring three-dimensional position coordinates of a surgical instrument used for surgery;
A first conversion matrix acquisition unit for acquiring a first conversion matrix for converting the coordinates of the image coordinate system set in the reconstructed image into coordinates of a coordinate system used by the three-dimensional coordinate acquisition unit for position coordinate measurement;
A video acquisition unit that acquires a two-dimensional moving image of the subject's operative field;
An image analysis unit for acquiring a two-dimensional position coordinate of the surgical instrument in a two-dimensional coordinate system set in the moving image acquired by the moving image acquisition unit;
Based on the three-dimensional position coordinates of the surgical tool acquired by the three-dimensional coordinate acquisition unit and the two-dimensional position coordinates of the surgical tool acquired by the image analysis unit, the three-dimensional position coordinates are obtained. A second transformation matrix acquisition unit that estimates a second transformation matrix to be transformed into the two-dimensional position coordinates;
An image superimposing unit that superimposes at least a part of the reconstructed image on the two-dimensional moving image as a two-dimensional image using the first transformation matrix and the second transformation matrix;
The image analysis unit analyzes the moving image acquired by the moving image acquisition unit to detect whether at least one of the angle of view or the line-of-sight direction of the moving image has been changed,
The surgery support device, wherein the second transformation matrix acquisition unit estimates the second transformation matrix when the image analysis unit detects a change in at least one of an angle of view or a line-of-sight direction. The image analysis unit
A background area specifying unit for specifying a background area for each frame of the moving image;
A difference image acquisition unit that acquires a difference image of a background image between different frames specified by the background region specifying unit;
The differential image acquisition unit includes a differential image analysis unit that acquires and analyzes the differential image to detect whether at least one of an angle of view or a line-of-sight direction of the moving image has been changed. Item 2. The surgical operation support device according to Item 1.   The position coordinate of the surgical tool acquired by the three-dimensional coordinate acquisition unit is a position coordinate measured based on a marker serving as a measurement reference provided in the surgical tool. The surgical operation support device described in 1. A program for realizing a surgery support function on a computer, wherein the surgery support function is:
A function of acquiring a reconstructed image of an area including a surgical part of a subject;
A function of acquiring a three-dimensional position coordinate of a surgical instrument used in surgery;
A function for acquiring a two-dimensional video image of the surgical field of the subject;
A function of acquiring a first conversion matrix for converting coordinates of an image coordinate system set in the reconstructed image into coordinates of a coordinate system used for measurement of three-dimensional position coordinates;
A function of acquiring a two-dimensional position coordinate of the surgical instrument in a two-dimensional coordinate system set in the moving image;
A function of detecting whether or not at least one of an angle of view or a line-of-sight direction of the two-dimensional moving image obtained by analyzing the moving image is changed;
Triggered by detecting a change in at least one of the angle of view or the line-of-sight direction, the three-dimensional position coordinates are determined based on the three-dimensional position coordinates and the two-dimensional position coordinates. A function of estimating a second transformation matrix for transformation into two-dimensional position coordinates;
A program comprising a function of superimposing at least a part of the reconstructed image on a two-dimensional moving image as a two-dimensional image using the first transformation matrix and the second transformation matrix. A three-dimensional measuring device for measuring the three-dimensional position coordinates of a surgical instrument used in surgery;
A microscope that captures the surgical field of the subject and generates a two-dimensional moving image;
An information processing device,
The information processing apparatus includes:
A medical image acquisition unit for acquiring a reconstructed image of a region including the surgical part of the subject;
A three-dimensional measurement unit that acquires three-dimensional position coordinates measured by the three-dimensional measurement device;
A moving image acquisition unit for acquiring a moving image generated by the microscope;
A first conversion matrix acquisition unit for acquiring a first conversion matrix for converting coordinates of the image coordinate system set in the reconstructed image into coordinates of a coordinate system used by the three-dimensional measurement unit for position coordinate measurement;
An image analysis unit for acquiring a two-dimensional position coordinate of the surgical instrument in a two-dimensional coordinate system set in the moving image generated by the microscope;
Based on the three-dimensional position coordinates of the surgical tool acquired by the three-dimensional measurement unit and the two-dimensional position coordinates of the surgical tool acquired by the image analysis unit, the three-dimensional position coordinates are calculated as described above. A second transformation matrix acquisition unit that estimates a second transformation matrix to be transformed into two-dimensional position coordinates;
An image superimposing unit that superimposes at least a part of the reconstructed image on a two-dimensional moving image as a two-dimensional image using the first transformation matrix and the second transformation matrix;
The image analysis unit analyzes the moving image captured by the moving image acquisition unit to detect whether at least one of the angle of view or the line-of-sight direction of the moving image acquisition unit is changed,
The surgery support system, wherein the second transformation matrix acquisition unit estimates the second transformation matrix when the image analysis unit detects a change in at least one of an angle of view or a line-of-sight direction.

Images