WO2022156425A1 - 一种微创手术器械定位方法和系统 - Google Patents

一种微创手术器械定位方法和系统 Download PDF

Info

Publication number
WO2022156425A1
WO2022156425A1 PCT/CN2021/137622 CN2021137622W WO2022156425A1 WO 2022156425 A1 WO2022156425 A1 WO 2022156425A1 CN 2021137622 W CN2021137622 W CN 2021137622W WO 2022156425 A1 WO2022156425 A1 WO 2022156425A1
Authority
WO
WIPO (PCT)
Prior art keywords
point
feature points
eye image
surgical instruments
point cloud
Prior art date
Application number
PCT/CN2021/137622
Other languages
English (en)
French (fr)
Inventor
熊璟
蔡伟鹏
夏泽洋
谢高生
刘勇
Original Assignee
中国科学院深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院 filed Critical 中国科学院深圳先进技术研究院
Publication of WO2022156425A1 publication Critical patent/WO2022156425A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • the present invention relates to the technical field of medical image processing, and more particularly, to a minimally invasive surgical instrument positioning method and system.
  • the ultimate purpose of surgical instrument positioning is to position the tip of the instrument.
  • the tip of the surgical instrument is in contact with the location of the disease, which makes it difficult to directly identify and mark the tip of the surgical instrument. Therefore, it is necessary to detect the surgical instrument through visual technology in the instrument.
  • the features of the surgical instrument are positioned by determining the three-dimensional coordinates of the features, so as to achieve the function of determining the position of the tip.
  • an identification is usually attached to a surgical instrument, for example, an active LED marker is installed on the surgical instrument, and the position of the surgical instrument is determined by recognizing the position of the marker in the field of view.
  • This additional identification requires modification of the surgical instrument, which increases the manufacturing cost of the surgical instrument and reduces the versatility of the technology.
  • the purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide a minimally invasive surgical instrument positioning method and system based on stereo vision, which solves the problems of limited visual field and lack of tactile perception in traditional minimally invasive surgery.
  • a method for positioning a minimally invasive surgical instrument includes the following steps:
  • the calculated 3D point cloud data and the standard point cloud model of surgical instruments are registered in the stereoscopic field of view to obtain the positioning information of surgical instruments.
  • a minimally invasive surgical instrument positioning system includes:
  • Surgical instrument segmentation unit used to input images containing surgical instruments into a pre-trained neural network model to segment out surgical instruments;
  • Feature point detection and matching unit used to detect the feature points of the segmented surgical instruments, and remove the wrongly matched feature points from the detected feature points, and then determine the significant feature points;
  • Three-dimensional coordinate calculation unit used to calculate the three-dimensional point cloud coordinates of the significant feature points according to the parallax data of the left-eye image and the right-eye image;
  • Positioning unit It is used to register the calculated 3D point cloud data and the standard point cloud model of surgical instruments in the stereoscopic field of view with the goal of minimizing the registration error function, so as to obtain the positioning information of the surgical instruments.
  • the present invention has the advantages that the distinctive feature points of the minimally invasive surgical instrument itself are detected through the binocular image under the stereo laparoscope, and the feature points detected by the binocular image are matched and calculated to eliminate false matching, and further The three-dimensional coordinate point cloud is calculated, and finally the calculated three-dimensional point cloud is registered with the standard point cloud model of minimally invasive surgical instruments in the stereoscopic field of view, and the positioning information of the entire surgical instrument in the stereoscopic laparoscopic field of view is obtained. Using the invention, the surgical instruments can be accurately identified and positioned.
  • FIG. 1 is a schematic diagram of a linear model of a binocular camera according to an embodiment of the present invention
  • FIG. 2 is a flowchart of a method for positioning a minimally invasive surgical instrument according to an embodiment of the present invention
  • FIG. 3 is a schematic process diagram of a minimally invasive surgical instrument positioning method according to an embodiment of the present invention.
  • the invention proposes a minimally invasive surgical instrument positioning scheme under the stereoscopic laparoscope, which aims to accurately detect the surgical instrument and accurately locate the position of the surgical instrument in a complex surgical environment.
  • the provided minimally invasive surgical instrument positioning method includes: segmentation of surgical instruments, feature point detection and matching of surgical instruments, three-dimensional coordinate calculation of feature points, three-dimensional feature point cloud and standard point cloud model of minimally invasive surgical instruments in Registration in the stereoscopic field of view, so as to realize the positioning of the instrument in the stereoscopic laparoscopic field of view.
  • the provided minimally invasive surgical instrument positioning method includes the following steps.
  • Step S110 segment the surgical instrument from the image containing the surgical instrument.
  • the present invention divides surgical instruments based on an improved full convolutional neural network (FCN), and uses the improved FCN network to perform training and segmentation of surgical instruments, so that surgical instruments can be segmented more accurately.
  • FCN full convolutional neural network
  • the surgical instrument segmentation algorithm mainly includes a network training stage and a surgical instrument segmentation stage.
  • the training phase the dataset produced by stereoscopic video is input into the improved FCN network, the network is learned by adopting the stochastic gradient descent method, and the FCN-8s network parameters are trained by fine-tuning, when the network loss converges to the equilibrium value , to stop network training.
  • the stage of surgical instrument segmentation real-time images are obtained mainly through video, and then the segmentation results of surgical instruments are obtained by calculating the trained network model.
  • the improved design of the present invention is to add a convolution operation to each of pool2 (pooling layer 2), pool3 (pooling layer 3), and pool4 (pooling layer 4), and then perform a fusion operation.
  • the added convolution process can extract deeper local information from the three pooling layers for surrounding information fusion, enhance the model's ability to judge local details, and optimize the detail effect of image segmentation.
  • each type of surgical instrument can be identified from a picture containing multiple types of surgical instruments.
  • the improved FCN can input images of surgical instruments of any size, and learn the network structure end-to-end to complete the training purpose. Further, the trained model can be used to segment surgical instruments in real time.
  • Step S120 detecting and matching the feature points of the surgical instrument to determine the significant feature points.
  • the ORB Oriented Fast and Rotated Brief
  • the main principle is to detect the pixel value of a circle around the candidate feature point based on the gray value of the image around the feature point. If there are enough pixels in the area around the candidate point and the gray value of the candidate point is sufficiently different, then Take the candidate point as a feature point. For example, for the candidate feature point p, if there are more than 8 connected pixels in the set domain range, which are darker or brighter than the candidate feature point p, then p is used as the detected feature point.
  • Feature points are used to label prominent points in an image, such as contour points, bright spots in darker areas, dark spots in brighter areas, etc.
  • ORB can shorten the time to search for feature points, and the selected feature points are located in areas with varying brightness, such areas can define the boundaries and regions of the surgical instrument, so these feature points can identify the surgical instrument, not in the image. any other object or background. In this way, the tip of the surgical instrument can also be precisely identified.
  • Step S121 detecting the feature points of the left and right view images and calculating the descriptors of the feature points
  • ORB For example, first use ORB to extract feature points and corresponding descriptors for two pictures L and R respectively.
  • Descriptors are used to represent the attributes of feature points. For example, in images with different sizes, orientations, and brightness, the same feature point should have sufficiently similar descriptors.
  • Step S122 match by BF (Brute Force) brute force matching algorithm
  • Step S123 dividing the image into G grids
  • Step S124 by calculating the correct matching number n and the threshold value near the feature point that the BF is matched to determine whether the point is correctly matched;
  • Step S130 Calculate the three-dimensional point cloud coordinates of the significant feature points.
  • the image captured by the camera is the process of projecting the 3D space point information to the imaging plane, and the 3D point cloud reprojection is to map the 2D pixel points in the imaging plane into the 3D space according to the parallax data.
  • the coordinates of the point P in the space imaged in the left and right camera image coordinate systems are (u l , v l ), (u r , v r ), respectively, u 0 , v 0 are the coordinate values of the image center position, and f is the focal length .
  • O l represents the left camera
  • Or represents the right camera.
  • Step S140 register the calculated three-dimensional point cloud with the standard point cloud model of the minimally invasive surgical instrument in the stereoscopic field of view.
  • s diag(s 1 ,s 2 ,...,s m ) is a scale matrix
  • r is a rotation matrix
  • t is a translation matrix
  • p i is the source point
  • q j is the target point
  • N m is the target point
  • N d is the number of source points.
  • the registration problem is further transformed into the problem of solving the optimal solution.
  • Im is the identity matrix
  • the implementation of the scaled ICP algorithm includes the following steps:
  • Step S141 establish a correlation through the current transformation (s k , r k , t k ), and calculate as follows:
  • Surgical instruments can be accurately positioned through point cloud registration, such as locating the tips of surgical instruments.
  • the present invention also provides a positioning system for a minimally invasive surgical instrument, which is used to implement one or more aspects of the above method.
  • the system includes: a surgical instrument segmentation unit for inputting images containing surgical instruments to a pre-trained neural network model to segment the surgical instruments; a feature point detection and matching unit for detecting the segmented surgery Device feature points, and remove the wrongly matched feature points from the detected feature points, and then determine the significant feature points; a three-dimensional coordinate calculation unit, which is used for calculating the three-dimensional point of the significant feature points according to the parallax data of the left-eye image and the right-eye image. Cloud coordinates; and a positioning unit, which is used to register the calculated three-dimensional point cloud data with the standard point cloud model of the surgical instrument in the stereoscopic field of view with the goal of minimizing the registration error function to obtain the positioning information of the surgical instrument.
  • the present invention utilizes the characteristics of binocular vision imaging to provide doctors with three-dimensional visual information, and through accurate positioning of surgical instruments, it overcomes the problem of limited visual field in traditional laparoscopic surgery, and improves the doctor's ability to perceive the intraoperative environment. .
  • the present invention does not require additional identification, and enables doctors to diagnose and operate more accurately by identifying the characteristics of the surgical instrument and registering the point cloud, reducing intraoperative risks and improving the success rate of the operation.
  • the present invention may be a system, method and/or computer program product.
  • the computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.
  • a computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Non-exhaustive list of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above.
  • RAM random access memory
  • ROM read only memory
  • EPROM erasable programmable read only memory
  • flash memory static random access memory
  • SRAM static random access memory
  • CD-ROM compact disk read only memory
  • DVD digital versatile disk
  • memory sticks floppy disks
  • mechanically coded devices such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above.
  • Computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.
  • the computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • the computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages.
  • Source or object code written in any combination including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages.
  • the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect).
  • LAN local area network
  • WAN wide area network
  • custom electronic circuits such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs)
  • FPGAs field programmable gate arrays
  • PDAs programmable logic arrays
  • Computer readable program instructions are executed to implement various aspects of the present invention.
  • These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
  • These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
  • Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions.
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Abstract

一种微创手术器械定位方法和系统。该方法：将包含手术器械的图像输入至预训练的神经网络模型，分割出手术器械；检测所分割出的手术器械特征点，并从检测出的特征点剔除错误匹配的特征点，进而确定显著特征点；根据左目图像和右目图像的视差数据计算所述显著特征点的三维点云坐标；以最小化配准误差函数为目标，将所计算的三维点云数据与手术器械标准点云模型在立体视野中配准，获得手术器械的定位信息。能够对手术器械进行精准识别和定位。

Description

一种微创手术器械定位方法和系统 技术领域
本发明涉及医学图像处理技术领域，更具体地，涉及一种微创手术器械定位方法和系统。
背景技术
由于传统腹腔镜手术中医生可观察的视野范围有限，绝大多数外科手术需要凭借医生的经验确定手术过程，而且其中有些病发部分是不可预见的，影响了手术效果。随着立体双目视觉技术在医疗领域的应用，基于立体视觉的微创手术器械定位可以用于手术导航，辅助医生分析手术器械与病发部分的位置与方向，这对于减少手术创伤和提高手术质量有重大意义，并对手术机器人技术的应用有重要价值。
作为一门新发展的微创方法，腹腔镜手术因为其术后瘢痕小、疼痛轻、恢复快、住院时间短等优势，深受患者青睐，成为越来越多患者的最佳选择，但是由于腹腔镜手术中存在视野局限性、医生对术中环境感知差，也造成了术中出血、病灶边界定位不准、健康组织切除过多、并发症风险高等诸多问题。计算机视觉技术可以进行微创手术器械定位，为医生提供更丰富的交互信息，辅助医生完成手术操作，提高手术质量。
手术器械定位的最终目的是进行器械尖端的定位，在手术过程中，手术器械的尖端与病发位置相接触，导致难以直接识别和标记手术器械尖端，所以需要在器械中通过视觉技术检测手术器械的特征，通过确定特征的三维坐标对手术器械尖端进行定位，以达到确定尖端位置的作用。
在现有技术中，通常在手术器械上附加标识，例如在手术器械上安装主动式LED标识物，通过识别视野中的标识物位置去确定手术器械的位置。这种附加标识需要对手术器械进行改造，增加了手术器械的制造成本并降低了技术的通用性。
发明内容
本发明的目的是克服上述现有技术的缺陷，提供一种基于立体视觉的微创手术器械定位方法和系统，解决了传统微创手术中存在的视野局限性和缺乏触觉感知的问题。
根据本发明的第一方面，提供一种微创手术器械定位方法。该方法包括以下步骤：
将包含手术器械的图像输入至预训练的神经网络模型，分割出手术器械；
检测所分割出的手术器械特征点，并从检测出的特征点剔除错误匹配的特征点，进而确定显著特征点；
根据左目图像和右目图像的视差数据计算所述显著特征点的三维点云坐标；
以最小化配准误差函数为目标，将所计算的三维点云数据与手术器械标准点云模型在立体视野中配准，获得手术器械的定位信息。
根据本发明的第二方面，提供一种微创手术器械定位系统。该系统包括：
手术器械分割单元：用于将包含手术器械的图像输入至预训练的神经网络模型，分割出手术器械；
特征点检测与匹配单元：用于检测所分割出的手术器械特征点，并从检测出的特征点剔除错误匹配的特征点，进而确定显著特征点；
三维坐标计算单元：用于根据左目图像和右目图像的视差数据计算所述显著特征点的三维点云坐标；
定位单元：用于以最小化配准误差函数为目标，将所计算的三维点云数据与手术器械标准点云模型在立体视野中配准，获得手术器械的定位信息。
与现有技术相比，本发明的优点在于，在立体腹腔镜下通过双目图像检测微创手术器械自身的显著特征点，对双目图像检测的特征点做匹配计算并消除误匹配，进而计算出三维坐标点云，最后将计算的三维点云与微创手术器械标准点云模型在立体视野做配准，获得手术器械整体在立体腹腔镜视野中的定位信息。利用本发明能够对手术器械进行精准识别和定位。
通过以下参照附图对本发明的示例性实施例的详细描述，本发明的其 它特征及其优点将会变得清楚。
附图说明
被结合在说明书中并构成说明书的一部分的附图示出了本发明的实施例，并且连同其说明一起用于解释本发明的原理。
图1是根据本发明一个实施例的双目相机线性模型的示意图；
图2是根据本发明一个实施例的微创手术器械定位方法的流程图；
图3是根据本发明一个实施例的微创手术器械定位方法的过程示意图。
具体实施方式
现在将参照附图来详细描述本发明的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本发明的范围。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本发明及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
在这里示出和讨论的所有例子中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它例子可以具有不同的值。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
本发明提出了一种立体腹腔镜下的微创手术器械定位方案，目的是在复杂的手术环境中准确检测手术器械并且准确定位手术器械的位置。简言之，所提供的微创手术器械定位方法包括：手术器械的分割、手术器械的特征点检测与匹配、特征点的三维坐标计算、三维特征点云与微创手术器械标准点云模型在立体视野中配准，从而实现在立体腹腔镜视野中的器械定位。
具体地，结合图1和图2所示，所提供的微创手术器械定位方法包括以下步骤。
步骤S110，从包含手术器械的图像中分割出手术器械。
精准的手术器械分割是辅助微创手术规划和疾病诊断的重要手段。在一个优选实施例中，本发明基于改进的全卷积神经网络(FCN)对手术器械进行分割，使用改进的FCN网络对手术器械进行训练分割，能够较精确的分割出手术器械。
具体地，手术器械分割算法主要包括网络训练阶段和手术器械分割阶段。在训练阶段将立体内镜视频制作的数据集输入到改进的FCN网络中，通过采用随机梯度下降法进行网络学习，并且通过微调对FCN-8s网络参数进行训练，当网络损失收敛到平衡值时，停止网络训练。手术器械分割阶段，主要是通过视频获取实时图像，然后利用训练好的网络模型计算得到手术器械的分割结果。
本发明的改进设计是在pool2(池化层2)、pool3(池化层3)、pool4(池化层4)各增加一次卷积操作，然后再进行融合操作。增加的卷积过程可以从三个池化层中提取到更深层的局部信息进行周围的信息融合，增强模型的局部细节判断能力，从而优化图像分割的细节效果。
通过对手术器械分割，能够从包含多种类型的手术器械图片中，识别出每种类型的手术器械。改进的FCN可输入任意大小的手术器械图像，以端到端的学习网络结构，完成训练目的。进一步，利用训练后的模型即可实时的分割手术器械。
步骤S120，检测并匹配手术器械的特征点，以确定显著特征点。
在一个实施例中，采用ORB(Oriented Fast and Rotated Brief)算法来检测手术器械的特征点。其主要原理是基于特征点周围的图像灰度值，检测候选特征点周围一圈的像素值，如果候选点周围领域内有足够多的像素点与该候选点的灰度值差别够大，则将该候选点作为一个特征点。例如，对于候选特征点p，如果设定的领域范围内有8个以上的相连像素，暗于或亮于该候选特征点p，则将p作为检测出的特征点。特征点用于标注图像中比较显著的点，如轮廓点，较暗区域中的亮点，较亮区域中的暗点等。
利用ORB能够缩短搜索特征点的时间，并且所选出的特征点位于亮度有变化的区域，此类区域能够定义手术器械的界限和区域，因此这些特征点 能够识别出手术器械，而不是图像中的任何其他对象或背景。通过这种方式，也能够精确识别手术器械尖端。
进一步地，在检测到特征点后，为了解决特征匹配问题，提高匹配的稳定性，需要从检测得到的特征点中，去除低对比度的特征点以及不稳定的边缘响应点。
在一个实施例中，使用一种运动统计特性的方法，可以迅速剔除错误的匹配，提高匹配的稳定性。主要流程如下：
步骤S121，检测左右视图图像特征点并计算特征点的描述子；
例如，先对两张图片L和R分别使用ORB提取特征点和对应的描述子。描述子用于表示特征点的属性。例如，在大小、方向、明暗不同的图像中，同一特征点应具有足够相似的描述子。
步骤S122，通过BF(Brute Force)暴力匹配算法进行匹配；
步骤S123，将图像划分成G个网格；
步骤S124，通过计算BF匹配好的特征点附近的正确匹配个数n与阈值来判断是否该点被正确匹配；
上述剔除错误匹配的基本思想是：
先对两张图片L和R分别使用ORB提取特征点和对应的描述子，设分别有N和M个；对两张图片的特征点进行BF匹配，这样能够找到图片L中每个特征点对应的R中的最邻近的特征点。由于运动的平滑性，在正确匹配的特征点附近的正确匹配点对数应该大于错误匹配的特征点附近的正确匹配点对数。通过这种方式，可以根据BF匹配好的特征点附近的正确匹配个数n与阈值来判断该点是否被正确匹配。通过BF匹配，能够尝试所有可能的匹配，从而能够找到最佳匹配。
步骤S130，计算显著特征点的三维点云坐标。
相机拍摄的图像是将三维空间点信息投影到成像平面的过程，而三维点云重投影是根据视差数据将成像平面中的二维像素点映射到三维空间中。
如图3的双目视觉的线性模型中的相似三角关系可以得到成像点坐标域点P的三维空间位置(x,y,z)间的关系式，表示为：
Figure PCTCN2021137622-appb-000001
其中，空间中的点P在左右相机图像坐标系成像的坐标分别为(u l,v l)、(u r,v r)，u 0,v 0是图像中心位置的坐标值，f是焦距。根据主视图中点的像素坐标以及这一点对应的视差d，可以求出此点在三维空间中的坐标，图中O l,表示左相机，O r表示右相机。
步骤S140，将计算的三维点云与微创手术器械标准点云模型在立体视野中的配准。
在实际配准过程中，需要考虑尺度配准问题，也就是既要考虑尺度变换又要考虑刚体变换，利用三维点云在刚性变换中，结合ICP(迭代最近点)算法，计算最小化配准误差函数，表示为：
Figure PCTCN2021137622-appb-000002
其中s＝diag(s 1,s 2,…,s m)是一个尺度矩阵，r是一个旋转矩阵，t是一个平移矩阵，p i是源点，q j是目标点，N m是目标点的数量，N d是源点的数目。
因此配准问题又进一步转换成求解最优解问题
Figure PCTCN2021137622-appb-000003
s.t r Tr＝I m,det(r)＝1
其中，I m是单位矩阵。
与ICP算法一样，带尺度ICP算法的实现包括以下步骤：
步骤S141、通过当前的变换(s k,r k,t k)建立相关性，计算如下：
Figure PCTCN2021137622-appb-000004
步骤S142)、令s＝diag(s 1,s 2,…,s m)，计算新的变换(s k+1,r k+1,t k+1)，计算式如下：
Figure PCTCN2021137622-appb-000005
重复以上步骤S141，S142，如果s的变化量Δs＝|s k+1-s k|小于阈值ε或者达到做大迭代次数，则停止迭代，其中k是迭代次数索引，
Figure PCTCN2021137622-appb-000006
表示源点与对应目标点之间的距离最小的目标点，c k+1(i)表示源点与对应目标点之 间的距离最小。
通过点云配准能够对手术器械进行准确定位，例如定位手术器械的尖端。
相应地，本发明还提供一种微创手术器械的定位系统，用于实现上述方法的一个方面或多个方面。例如，该系统包括：手术器械分割单元，其用于将包含手术器械的图像输入至预训练的神经网络模型，分割出手术器械；特征点检测与匹配单元，其用于检测所分割出的手术器械特征点，并从检测出的特征点剔除错误匹配的特征点，进而确定显著特征点；三维坐标计算单元，其用于根据左目图像和右目图像的视差数据计算所述显著特征点的三维点云坐标；定位单元，其用于以最小化配准误差函数为目标，将所计算的三维点云数据与手术器械标准点云模型在立体视野中配准，获得手术器械的定位信息。
经验证，本发明利用双目视觉成像特点，为医生提供三维的视觉信息，通过对手术器械的准确定位，克服传统腹腔镜手术中的视野局限性问题，提升了医生对术中环境的感知能力。并且，本发明无需附加标识，通过对手术器械自身特征的识别与点云配准，使医生更精确地诊断和手术，降低术中风险，提高手术成功率。
本发明可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本发明的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬 时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本发明操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++、Python等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本发明的各个方面。
这里参照根据本发明实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本发明的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它 可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本发明的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是，通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。
以上已经描述了本发明的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进，或者使本技术领域的其它普通技术人员能理解 本文披露的各实施例。本发明的范围由所附权利要求来限定。

Claims (10)

  1. 一种微创手术器械定位方法，包括以下步骤：
    将包含手术器械的图像输入至预训练的神经网络模型，分割出手术器械；
    检测所分割出的手术器械特征点，并从检测出的特征点剔除错误匹配的特征点，进而确定显著特征点；
    根据左目图像和右目图像的视差数据计算所述显著特征点的三维点云坐标；
    以最小化配准误差函数为目标，将所计算的三维点云数据与手术器械标准点云模型在立体视野中配准，获得手术器械的定位信息。
  2. 根据权利要求1所述的方法，其特征在于，根据以下步骤检测特征点：
    设定候选特征点的领域范围，计算所述领域范围内各像素点与该候选特征点的像素值的差值；
    在所述领域范围内计算所述差值大于设定阈值的像素点数目，如该像素点数据大于设定的次数阈值，则认为所述候选特征点是检测出的特征点。
  3. 根据权利要求2所述的方法，其特征在于，根据以下步骤从检测出的特征点剔除错误匹配的特征点：
    分别检测左目图像和右目图像的特征点和对应的描述子，设分别有N和M个；
    对左目图像和右目图像的特征点进行BF匹配，找到左目图像中每个特征点对应的右目图像中的最邻近的特征点；
    通过比较BF匹配好的特征点附近的正确匹配个数与阈值来判断各点是否被正确匹配，以剔除错误匹配的特征点。
  4. 根据权利要求1所述的方法，其特征在于，所述神经网络模型是多尺度全卷积神经网络。
  5. 根据权利要求1所述的方法，其特征在于，对于成像点坐标域点P， 根据以下换算公式计算三维点云坐标位置(x,y,z)：
    Figure PCTCN2021137622-appb-100001
    其中，空间中的点P在左右相机图像坐标系成像的坐标分别为(u l,v l)、(u r,v r)，u 0,v 0是图像中心位置的坐标值，f是焦距，d是主视图中点的像素坐标以及这一点对应的视差。
  6. 根据权利要求1所述的方法，其特征在于，所述最小化配准误差函数表示为：
    Figure PCTCN2021137622-appb-100002
    或转换成求解最优解问题：
    Figure PCTCN2021137622-appb-100003
    s.t  r Tr＝I m,det(r)＝1
    其中，s＝diag(s 1,s 2,…,s m)是一个尺度矩阵，r是一个旋转矩阵，t是一个平移矩阵，p i是源点，q j是目标点，N m是目标点的数量，N d是源点的数目。
  7. 根据权利要求6所述的方法，其特征在于，根据以下步骤求解所述最小化配准误差函数：
    步骤S71，通过当前的变换(s,r k,t k)建立相关性，计算如下：
    Figure PCTCN2021137622-appb-100004
    步骤S72，令s＝diag(s 1,s 2,…,s m)，计算新的变换(s k+1,r k+1,t k+1)，计算式如下
    Figure PCTCN2021137622-appb-100005
    步骤S73，重复步骤S71和S72，直到s的变化量Δs＝|s k+1-s k|小于阈值ε或者迭代次数k达到设定阈值；
    其中k是迭代次数索引，
    Figure PCTCN2021137622-appb-100006
    表示源点与对应目标点之间的距离最小的目标点，c k+1(i)表示源点与对应目标点之间的距离最小。
  8. 一种微创手术器械定位系统，包括：
    手术器械分割单元：用于将包含手术器械的图像输入至预训练的神经网络模型，分割出手术器械；
    特征点检测与匹配单元：用于检测所分割出的手术器械特征点，并从检测出的特征点剔除错误匹配的特征点，进而确定显著特征点；
    三维坐标计算单元：用于根据左目图像和右目图像的视差数据计算所述显著特征点的三维点云坐标；
    定位单元：用于以最小化配准误差函数为目标，将所计算的三维点云数据与手术器械标准点云模型在立体视野中配准，获得手术器械的定位信息。
  9. 一种计算机可读存储介质，其上存储有计算机程序，其中，该程序被处理器执行时实现根据权利要求1至7中任一项所述方法的步骤。
  10. 一种计算机设备，包括存储器和处理器，在所述存储器上存储有能够在处理器上运行的计算机程序，其特征在于，所述处理器执行所述程序时实现权利要求1至7中任一项所述的方法的步骤。
PCT/CN2021/137622 2021-01-21 2021-12-13 一种微创手术器械定位方法和系统 WO2022156425A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110084081.8 2021-01-21
CN202110084081.8A CN112734776B (zh) 2021-01-21 2021-01-21 一种微创手术器械定位方法和系统

Publications (1)

Publication Number Publication Date
WO2022156425A1 true WO2022156425A1 (zh) 2022-07-28

Family

ID=75594839

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/137622 WO2022156425A1 (zh) 2021-01-21 2021-12-13 一种微创手术器械定位方法和系统

Country Status (2)

Country Link
CN (1) CN112734776B (zh)
WO (1) WO2022156425A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115641365A (zh) * 2022-08-25 2023-01-24 北京微视威信息科技有限公司 点云配准方法、系统、设备及介质
CN116350352A (zh) * 2023-02-23 2023-06-30 北京纳通医用机器人科技有限公司 手术机器人标志位识别定位方法、装置及设备
CN116563356A (zh) * 2023-05-12 2023-08-08 北京长木谷医疗科技股份有限公司 全局3d配准方法、装置及电子设备

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112734776B (zh) * 2021-01-21 2023-04-18 中国科学院深圳先进技术研究院 一种微创手术器械定位方法和系统
CN115205356B (zh) * 2022-09-08 2022-12-30 广州易道智慧信息科技有限公司 一种基于双目立体视觉的实训平台快速调试方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150138186A1 (en) * 2012-05-18 2015-05-21 Cydar Limited Virtual fiducial markers
CN106952347A (zh) * 2017-03-28 2017-07-14 华中科技大学 一种基于双目视觉的超声手术辅助导航系统
CN111652919A (zh) * 2020-07-01 2020-09-11 杭州脉流科技有限公司 基于左心室三维模型配准定位长短轴的方法、装置、计算机设备和存储介质
CN112037268A (zh) * 2020-09-02 2020-12-04 中国科学技术大学 一种动态场景下的基于概率传递模型的环境感知方法
CN112734776A (zh) * 2021-01-21 2021-04-30 中国科学院深圳先进技术研究院 一种微创手术器械定位方法和系统

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ITTV20100133A1 (it) * 2010-10-08 2012-04-09 Teleios Srl Apparato e metodo per effettuare la mappatura di uno spazio tridimensionale in applicazioni medicali a scopo interventistico o diagnostico

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150138186A1 (en) * 2012-05-18 2015-05-21 Cydar Limited Virtual fiducial markers
CN106952347A (zh) * 2017-03-28 2017-07-14 华中科技大学 一种基于双目视觉的超声手术辅助导航系统
CN111652919A (zh) * 2020-07-01 2020-09-11 杭州脉流科技有限公司 基于左心室三维模型配准定位长短轴的方法、装置、计算机设备和存储介质
CN112037268A (zh) * 2020-09-02 2020-12-04 中国科学技术大学 一种动态场景下的基于概率传递模型的环境感知方法
CN112734776A (zh) * 2021-01-21 2021-04-30 中国科学院深圳先进技术研究院 一种微创手术器械定位方法和系统

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115641365A (zh) * 2022-08-25 2023-01-24 北京微视威信息科技有限公司 点云配准方法、系统、设备及介质
CN115641365B (zh) * 2022-08-25 2023-08-04 北京微视威信息科技有限公司 点云配准方法、系统、设备及介质
CN116350352A (zh) * 2023-02-23 2023-06-30 北京纳通医用机器人科技有限公司 手术机器人标志位识别定位方法、装置及设备
CN116350352B (zh) * 2023-02-23 2023-10-20 北京纳通医用机器人科技有限公司 手术机器人标志位识别定位方法、装置及设备
CN116563356A (zh) * 2023-05-12 2023-08-08 北京长木谷医疗科技股份有限公司 全局3d配准方法、装置及电子设备

Also Published As

Publication number Publication date
CN112734776B (zh) 2023-04-18
CN112734776A (zh) 2021-04-30

Similar Documents

Publication Publication Date Title
WO2022156425A1 (zh) 一种微创手术器械定位方法和系统
Lin et al. Video‐based 3D reconstruction, laparoscope localization and deformation recovery for abdominal minimally invasive surgery: a survey
Collins et al. Augmented reality guided laparoscopic surgery of the uterus
CN108369741B (zh) 用于配准数据的方法和系统
Glocker et al. Real-time RGB-D camera relocalization
US20180174311A1 (en) Method and system for simultaneous scene parsing and model fusion for endoscopic and laparoscopic navigation
Grasa et al. Visual SLAM for handheld monocular endoscope
Shen et al. Context-aware depth and pose estimation for bronchoscopic navigation
US20150297313A1 (en) Markerless tracking of robotic surgical tools
Luó et al. Development and comparison of new hybrid motion tracking for bronchoscopic navigation
KR20180087947A (ko) 3차원의 포인트 클라우드를 이용한 모델링 방법 및 모델링 장치
US20110282151A1 (en) Image-based localization method and system
Petersen et al. Real-time modeling and tracking manual workflows from first-person vision
Mountney et al. A probabilistic framework for tracking deformable soft tissue in minimally invasive surgery
Collins et al. Robust, real-time, dense and deformable 3D organ tracking in laparoscopic videos
Chen et al. Real‐time geometry‐aware augmented reality in minimally invasive surgery
JP7147753B2 (ja) 情報処理装置、情報処理方法、及びプログラム
Sánchez-González et al. Laparoscopic video analysis for training and image-guided surgery
Prisacariu et al. Shared shape spaces
WO2022170562A1 (zh) 一种消化内镜导航方法和系统
Wang et al. A visual SLAM-based bronchoscope tracking scheme for bronchoscopic navigation
Su et al. Comparison of 3d surgical tool segmentation procedures with robot kinematics prior
Lee et al. Weakly supervised segmentation for real‐time surgical tool tracking
KR20150082417A (ko) 병렬화가능한 구조에서의 이미지들을 사용하여 표면 엘리먼트들의 로컬 지오메트리 또는 표면 법선들을 초기화 및 솔빙하는 방법
Wang et al. Depth-based branching level estimation for bronchoscopic navigation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21920799

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21920799

Country of ref document: EP

Kind code of ref document: A1