CN115245303A - Image fusion system and method for endoscope three-dimensional navigation - Google Patents

Abstract

The invention discloses an image fusion system and method for endoscope three-dimensional navigation. The system comprises a graphic workstation, an optical positioning mark, an optical positioning system and an endoscope system, wherein the graphic workstation is used for importing a three-dimensional image before examination, reconstructing and post-processing the three-dimensional image, extracting image point cloud data and optical point cloud data of a focus and a surrounding tissue structure of the focus, matching the image point cloud data and the optical point cloud data based on a point cloud registration algorithm, and calculating spatial transformation from a coordinate system of the optical positioning system to a coordinate system of the three-dimensional image before examination; calculating the space transformation from an endoscope viewing angle coordinate system to a three-dimensional image coordinate system before examination; and displaying the current visual field of the three-dimensional image corresponding to the endoscope image in real time based on the real-time space position of the endoscope, and overlapping and fusing the current visual field with the endoscope image for display. By utilizing the invention, the two-dimensional image of the endoscope field of vision is combined with the three-dimensional image before examination, and abundant image information is provided for the doctor to carry out the endoscopy for assisting the diagnosis.

Description

An image fusion system and method for endoscope three-dimensional navigation

technical field

The present invention relates to the technical field of endoscopic images, and in particular, to an image fusion system and method for endoscopic three-dimensional navigation.

Background technique

Three-dimensional image data such as CT/MR provides rich anatomical information of the patient's body tissue, including lesions and their spatial relationship with surrounding tissue structures such as blood vessels. In the medical image-guided interventional treatment process, the doctor determines the relative spatial relationship between the lesion and the surrounding tissue blood vessels by referring to the three-dimensional image of the patient, so as to avoid harming the important anatomical tissue structure of the patient during the inspection and treatment process.

Endoscope is a detection instrument that integrates traditional optics, ergonomics, precision machinery, modern electronics, mathematics, and software. It can enter the stomach through the mouth or enter the body through other natural orifices. As a result, lesions that cannot be shown by X-rays can be seen. With the help of endoscopy, doctors can observe, for example, ulcers or tumors in the stomach, and based on this, they can formulate the best treatment plan for the patient.

However, due to the limited field of view of the endoscope, it is difficult to see the whole picture of the lesion clearly, and it is impossible to accurately distinguish the spatial relationship between the lesion and the surrounding tissue and blood vessels. In addition, because the endoscope is in the patient's body, the doctor cannot accurately distinguish its current specific position in the patient's body. , which may introduce errors in the localization of the lesions.

SUMMARY OF THE INVENTION

In view of the above technical problems in the prior art, the present invention provides an image fusion system for 3D navigation of an endoscope, which can fuse and display in real time three-dimensional images such as CT/MR before examination and an image of the current field of view of the endoscope. By installing optical positioning markers on the endoscope to locate its spatial position in real time, the doctor uses the multi-angle scanning of the endoscope to view the lesion and its surrounding tissue structure, and the software uses multiple frames of endoscopic images to reconstruct the lesion and its surrounding tissue structure. The spatial point cloud is registered with the point cloud that has been segmented and reconstructed in advance in 3D images such as CT/MR to complete the registration of the human body space and the preoperative 3D image space. Then, by moving the endoscope in real time, the software provides the current endoscope. The three-dimensional image in the field of view of the endoscope is displayed superimposed with the endoscope image.

An embodiment of the present invention provides an image fusion system for endoscope three-dimensional navigation, including a graphics workstation, an optical positioning marker, an optical positioning system, and an endoscope system, wherein the graphics workstation is used to import pre-examination three-dimensional images and perform Reconstruction and post-processing, extract the image point cloud data of the lesion and its surrounding tissue structure, extract the optical point cloud data of the lesion and its surrounding tissue structure in the optical positioning space, and match the image point cloud data and optical points based on the point cloud registration algorithm Cloud data, calculate the spatial transformation from the optical positioning system coordinate system to the pre-examination 3D image coordinate system; calculate the spatial transformation from the endoscope viewing angle coordinate system to the pre-examination 3D image coordinate system; real-time display based on the real-time spatial position of the endoscope The current field of view of the three-dimensional image corresponding to the endoscopic image is superimposed and fused with the endoscopic image; the optical positioning mark is installed on the endoscope; the optical positioning system is used to track the spatial position of the optical positioning mark in real time; the endoscope system for endoscopy.

The embodiment of the present invention also provides an image fusion method for three-dimensional navigation of an endoscope, including: importing a pre-examination three-dimensional image into a graphics workstation; the graphics workstation divides the lesion and its surrounding tissue structure, and outputs a two-dimensional image of the segmentation result, Convert the two-dimensional image into a contour-only triangular patch, and further extract the image point cloud data of the lesion and its surrounding tissue structure; install an optical positioning marker on the endoscope, and set an optical positioning system to track the space of the optical positioning marker in real time position; use the endoscope to capture multiple frames of two-dimensional visual field images of the lesion and its surrounding tissue structure; based on the multiple frames of two-dimensional visual field images, the graphics workstation reconstructs the multiple frames of two-dimensional visual field images into the optical positioning space associated with the optical positioning markers. 3D point cloud data, extract the optical point cloud data of the lesion and its surrounding tissue structure in the optical positioning space; the graphics workstation matches the image point cloud data and the optical point cloud data based on the point cloud registration algorithm, and calculates the coordinate system of the optical positioning system to The spatial transformation of the three-dimensional image coordinate system before the inspection; the graphics workstation calculates the spatial transformation from the endoscope's viewing angle coordinate system to the three-dimensional image coordinate system before the inspection; moving the endoscope, based on the real-time spatial position of the endoscope, the graphics workstation displays in real time corresponding to The current field of view of the three-dimensional image of the endoscopic image is superimposed and fused with the endoscopic image.

In some embodiments, the optical positioning marks are reflective balls or black and white checkerboards for optical positioning.

In some embodiments, based on the spatial transformation of the endoscope viewing angle coordinate system to the optical positioning marker coordinate system, the spatial transformation of the optical positioning marker coordinate system to the optical positioning system coordinate system, and the optical positioning system coordinate system to the pre-examination three-dimensional image coordinate system to obtain the spatial transformation from the coordinate system of the endoscope viewing angle to the coordinate system of the three-dimensional image before the inspection.

Compared with the prior art, the beneficial effects of the embodiments of the present invention are:

1. Combine the two-dimensional image of the endoscopic field of view with the three-dimensional image before the examination to provide rich image information for doctors to assist in the diagnosis of endoscopic examination;

2. The 3D image before the examination can provide a clear and complete 3D map of the lesion and the surrounding tissue structure, which is convenient for doctors to observe the relative positional relationship between the lesion and the surrounding tissue structure in three-dimensional space from multiple angles in real-time during endoscopy;

3. Based on the point cloud registration algorithm, the registration of the human body space and the preoperative 3D image space is automatically completed, saving manual registration time.

Description of drawings

FIG. 1 is a structural block diagram of an image fusion system for endoscopic three-dimensional navigation according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of spatial transformation of each coordinate system.

FIG. 3 is a flowchart of a method for reconstructing a three-dimensional scene based on an endoscopic image according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of superimposed display of an endoscopic image and a three-dimensional image of a current field of view.

Detailed ways

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the accompanying drawings, but the present invention is not limited thereto. These and other features of the present invention will become apparent from the description of preferred forms of embodiment, given as non-limiting examples, with reference to the accompanying drawings.

The embodiment of the present invention provides an image fusion system for three-dimensional navigation of an endoscope.

FIG. 1 is a structural block diagram of an image fusion system for endoscopic three-dimensional navigation according to an embodiment of the present invention.

The image fusion system used for endoscope three-dimensional navigation includes a graphics workstation, an optical positioning marker, an optical positioning system, and an endoscope system.

The graphics workstation is used to import the pre-examination 3D image and perform reconstruction and post-processing, extract the image point cloud data of the lesion and its surrounding tissue structure, and extract the optical point cloud data of the lesion and its surrounding tissue structure in the optical positioning space, based on the point cloud The registration algorithm matches the image point cloud data and the optical point cloud data, and calculates the spatial transformation from the optical positioning system coordinate system to the pre-examination 3D image coordinate system; calculates the spatial transformation from the endoscope viewing angle coordinate system to the pre-examination 3D image coordinate system; Based on the real-time spatial position of the endoscope, the current field of view of the three-dimensional image corresponding to the endoscope image is displayed in real time, and is displayed in superposition and fusion with the endoscope image.

The optical positioning marker is installed on the endoscope, for example, a reflective ball or a black and white checkerboard can be installed at the end of the ureteroscope for optical positioning.

The optical positioning system is used to track the spatial position of the optical positioning marker in real time. For example, a binocular camera can be used to track the spatial position of an optical positioning marker such as a reflective ball or a black and white checkerboard mounted on the endoscope in real time based on the binocular positioning principle.

Endoscopy systems are used for routine endoscopy. The endoscope system is conventional, so it will not be repeated here.

FIG. 3 is a flowchart of an image fusion method for endoscopic three-dimensional navigation according to an embodiment of the present invention. The steps of the image fusion method for endoscope three-dimensional navigation of the present invention are:

Step S100, importing the pre-examination three-dimensional image.

In this step, import the three-dimensional image data of the patient before examination into the graphics workstation, such as CT image, MR image, three-dimensional ultrasound image, PET/CT image, and the like.

Step S200, the graphics workstation can automatically segment the lesion and its surrounding tissue structure based on algorithms such as threshold segmentation, active contour, sparse field level set, and deep learning, output a two-dimensional image of the segmentation result, and use algorithms such as isosurface extraction to convert the two-dimensional image. Convert to contour-only triangular patches, and further extract the image point cloud data of the lesion and its surrounding tissue structure;

In this step, the graphics workstation segments the lesion and its surrounding tissue structure, and extracts the image point cloud data D _I .

Step S300, an optical positioning mark is installed on the endoscope, and an optical positioning system is set to track the spatial position of the optical positioning mark in real time.

The optical positioning mark can be a reflective ball coated with a special material on the surface, or it can be a traditional flat black and white checkerboard. The optical positioning mark can be installed at the end of the endoscope. It is necessary to ensure that the optical positioning mark and the internal The relative spatial position relationship of the speculum remains unchanged, and the optical positioning mark can always be tracked and positioned by the optical positioning system;

Step S400, using an endoscope to capture multiple frames of two-dimensional visual field images of the lesion and its surrounding tissue structures.

In this step, the doctor uses an endoscope to scan from multiple angles to view the lesion and its surrounding tissue structure. This step is conventional, and thus will not be repeated here.

Step S500, based on the multi-frame two-dimensional field of view images, the graphics workstation reconstructs the multi-frame two-dimensional field of view images into three-dimensional point cloud data of the optical positioning space associated with the optical positioning mark, and extracts the lesion and its surrounding tissue structure in the optical positioning space. Optical point cloud data.

Based on the multi-frame 2D field of view images, the graphics workstation can automatically reconstruct the multi-frame 2D field of view images into 3D point cloud data in the optical positioning space based on the weighted iterative feature algorithm. Automatically segment the lesion and its surrounding tissue structure, and extract the optical point cloud data DO of the _lesion and its surrounding tissue structure in the optical positioning space;

The difference between optical point cloud data and image point cloud data is that the latter is generated based on the pre-examination 3D image, that is, the coordinate position information of the lesion and its surrounding tissue structures in the pre-examination 3D image space, while the former is generated by the optical point cloud data. The multi-frame endoscopic two-dimensional visual field images in the positioning space are generated, that is, the coordinate position information of the lesion and its surrounding tissue structures in the optical positioning space.

Step S600, the graphics workstation automatically matches the image point cloud data and the optical point cloud data based on point cloud registration algorithms such as sampling consistency initial registration and iterative closest point, and calculates the space from the coordinate system of the optical positioning system to the coordinate system of the three-dimensional image before the inspection. transform;

In this step, based on the point cloud registration algorithm, the registration of the image point cloud data D _I and the optical point cloud data D _O is automatically completed, the unification of the optical positioning space and the three-dimensional image space is realized, and the coordinate system of the optical positioning system is calculated to check. Spatial Transformation of Front 3D Image Coordinate System

便可求出其在三维图像空间中的坐标(x_I,y_I,z_I,1)^T。Where R is the rotation matrix of dimension 3x3, representing the attitude transformation between the two coordinate systems, which is a unit orthogonal matrix, and t is the translation vector of dimension 3x1, representing the translation transformation between the two coordinate systems. A point (x _o , y _o , z _o ) ^T in the optical positioning space, and its homogeneous coordinates (x _o , y _o , z _o , 1) ^T are transformed by left-multiplying space

The coordinates (x _I , y _I , z _I , 1) ^T in the three-dimensional image space can be obtained.

Step S700, the graphics workstation calculates the spatial transformation from the coordinate system of the viewing angle of the endoscope to the coordinate system of the three-dimensional image before the inspection.

表示内窥镜视角坐标系到光学定位标记坐标系的空间变换，通过机械设计尺寸获得，为已知；

表示光学定位标记坐标系到光学定位系统坐标系的空间变换，通过光学定位系统的定位信息获得，为已知；

表示光学定位系统坐标系到检查前三维图像坐标系的空间变换，由步骤S600获得；则内窥镜视角坐标系到检查前三维图像坐标系的空间变换

的计算公式为：The spatial transformation involved in this system is shown in Figure 2, and the symbol T represents the homogeneous matrix of the spatial transformation. in,

Represents the spatial transformation from the coordinate system of the endoscope viewing angle to the coordinate system of the optical positioning mark, which is obtained through the mechanical design size and is known;

Represents the spatial transformation of the optical positioning marker coordinate system to the optical positioning system coordinate system, obtained through the positioning information of the optical positioning system, and is known;

Represents the spatial transformation from the coordinate system of the optical positioning system to the coordinate system of the three-dimensional image before the inspection, which is obtained by step S600; then the spatial transformation from the coordinate system of the endoscope viewing angle to the coordinate system of the three-dimensional image before the inspection

The calculation formula is:

Step 800, moving the endoscope, based on the real-time spatial position of the endoscope, the graphics workstation displays the current field of view of the three-dimensional image corresponding to the endoscope image in real time, and superimposes and fuses the display with the endoscope image.

In this step, the endoscope is moved, and the graphics workstation displays the current field of view of the three-dimensional image corresponding to the endoscope image in real time, and superimposes and fuses the display with the endoscope image.

In this way, by using the system and method of the present invention, the two-dimensional image of the endoscopic field of view can be combined with the three-dimensional image before examination, so as to provide rich image information for doctors to assist in the diagnosis of endoscopic examination; the three-dimensional image before examination can provide The clear and complete 3D map of the lesion and the surrounding tissue structure is convenient for doctors to observe the relative positional relationship between the lesion and the surrounding tissue structure in three-dimensional space in real time during endoscopy; based on the point cloud registration algorithm, the human space and the preoperative three-dimensional image are automatically completed. Spatial registration saves manual registration time.

The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) may be used in combination with each other. For example, other embodiments may be utilized by those of ordinary skill in the art upon reading the above description. Additionally, in the foregoing Detailed Description, various features may be grouped together to simplify the present invention. This should not be construed as an intention that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description by way of example or embodiment, with each claim standing on its own as a separate embodiment, and it is contemplated that these embodiments may be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

The above embodiments are only exemplary embodiments of the present invention, and are not intended to limit the present invention, and the protection scope of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements to the present invention within the spirit and protection scope of the present invention, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present invention.

Claims (6)

1. an image fusion system for endoscope three-dimensional navigation, is characterized in that, comprises graphics workstation, optical positioning mark, optical positioning system, endoscope system, The graphics workstation is used to import the pre-examination 3D image and perform reconstruction and post-processing, extract the image point cloud data of the lesion and its surrounding tissue structure, and extract the optical point cloud data of the lesion and its surrounding tissue structure in the optical positioning space, based on the point cloud The registration algorithm matches the image point cloud data and the optical point cloud data, and calculates the spatial transformation from the optical positioning system coordinate system to the pre-examination 3D image coordinate system; calculates the spatial transformation from the endoscope viewing angle coordinate system to the pre-examination 3D image coordinate system; Based on the real-time spatial position of the endoscope, the current field of view of the three-dimensional image corresponding to the endoscope image is displayed in real time, and the display is superimposed and fused with the endoscope image; Optical positioning marks are installed on the endoscope; The optical positioning system is used to track the spatial position of the optical positioning marker in real time; Endoscopy systems are used for endoscopy. 2 . The image fusion system for endoscope three-dimensional navigation according to claim 1 , wherein the optical positioning mark is a reflective ball or a black and white checkerboard for optical positioning. 3 . 3 . The image fusion method for three-dimensional navigation of endoscope according to claim 1 , wherein, based on the spatial transformation of the endoscope viewing angle coordinate system to the optical positioning marker coordinate system, and the optical positioning marker coordinate system to the optical positioning The spatial transformation of the coordinate system of the system and the spatial transformation of the coordinate system of the optical positioning system to the coordinate system of the three-dimensional image before the inspection is used to obtain the spatial transformation of the coordinate system of the viewing angle of the endoscope to the coordinate system of the three-dimensional image before the inspection. 4. An image fusion method for endoscope three-dimensional navigation, characterized in that, comprising: Import the pre-examination 3D images into the graphics workstation; The graphics workstation segments the lesion and its surrounding tissue structure, outputs a two-dimensional image of the segmentation result, converts the two-dimensional image into a triangular patch with only contours, and further extracts the image point cloud data of the lesion and its surrounding tissue structure; Install an optical positioning marker on the endoscope, and set an optical positioning system to track the spatial position of the optical positioning marker in real time; Use endoscope to capture multiple frames of two-dimensional field images of the lesion and its surrounding tissue structure; Based on the multi-frame 2D visual field images, the graphics workstation reconstructs the multi-frame 2D visual field images into 3D point cloud data in the optical positioning space, and extracts the optical point cloud data of the lesion and its surrounding tissue structures in the optical positioning space; The graphics workstation matches the image point cloud data and the optical point cloud data based on the point cloud registration algorithm, and calculates the spatial transformation from the coordinate system of the optical positioning system to the coordinate system of the three-dimensional image before inspection; The graphics workstation calculates the spatial transformation from the coordinate system of the endoscope viewing angle to the coordinate system of the three-dimensional image before the inspection; When the endoscope is moved, based on the real-time spatial position of the endoscope, the graphics workstation displays the current field of view of the three-dimensional image corresponding to the endoscope image in real time, which is superimposed and fused with the endoscope image. 5 . The image fusion method for three-dimensional navigation of an endoscope according to claim 4 , wherein the optical positioning mark is a reflective ball or a black and white checkerboard for optical positioning. 6 . 6. The image fusion method for endoscope three-dimensional navigation according to claim 4, characterized in that, based on the spatial transformation of the endoscope viewing angle coordinate system to the optical positioning mark coordinate system, and the optical positioning mark coordinate system to the optical positioning The spatial transformation of the coordinate system of the system and the spatial transformation of the coordinate system of the optical positioning system to the coordinate system of the three-dimensional image before the inspection is used to obtain the spatial transformation of the coordinate system of the viewing angle of the endoscope to the coordinate system of the three-dimensional image before the inspection.

Images