WO2020253280A1 - Augmented reality navigation method and system for minimally invasive total knee replacement surgery - Google Patents

Augmented reality navigation method and system for minimally invasive total knee replacement surgery Download PDF

Info

Publication number
WO2020253280A1
WO2020253280A1 PCT/CN2020/079316 CN2020079316W WO2020253280A1 WO 2020253280 A1 WO2020253280 A1 WO 2020253280A1 CN 2020079316 W CN2020079316 W CN 2020079316W WO 2020253280 A1 WO2020253280 A1 WO 2020253280A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinate system
pose
virtual
relationship
hololens
Prior art date
Application number
PCT/CN2020/079316
Other languages
French (fr)
Chinese (zh)
Inventor
王君臣
王力
王田苗
Original Assignee
北京航空航天大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京航空航天大学 filed Critical 北京航空航天大学
Publication of WO2020253280A1 publication Critical patent/WO2020253280A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2055—Optical tracking systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2068—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis using pointers, e.g. pointers having reference marks for determining coordinates of body points

Definitions

  • the core problem that needs to be solved is how to unify the virtual scene space, intraoperative reality scene space and preoperative image space, so that the virtual anatomical model from the patient’s preoperative CT/MRI scan can be accurately The position and posture of is superimposed under the field of vision of the doctor wearing HoloLens.
  • the augmented reality navigation system for minimally invasive total knee replacement surgery may also have the following additional technical features:
  • Fig. 1 is a flowchart of an augmented reality navigation method for minimally invasive total knee replacement surgery according to an embodiment of the present invention
  • the augmented reality navigation method for minimally invasive total knee replacement surgery includes the following steps:
  • step S1 includes: using a binocular camera and a visual marker to assist in calibrating the HoloLens virtual scene space coordinate system.

Landscapes

  • Health & Medical Sciences (AREA)
  • Surgery (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Robotics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Disclosed are an augmented reality navigation method and system for minimally invasive total knee replacement surgery. The method comprises: obtaining a first relationship between a world coordinate system of a virtual space corresponding to a HoloLens application program and a coordinate system of a real scene; matching, on the basis of spatial transformation, an intraoperative knee joint point cloud with a preset 3D model point cloud, to obtain a second relationship between a preoperative medical image spatial coordinate system and a binocular camera coordinate system; and superimposing, according to the first relationship and the second relationship, virtual femur and tibia models and a corresponding surgical guide model under a HoloLens view to realize augmented reality navigation. This method can realize semi-automatic calibration of the HoloLens virtual space coordinate system, and combines image registration technology to accurately superimpose a virtual knee joint anatomical model and the virtual surgical guide model to a corresponding real affected position, thereby providing a doctor with intuitive and accurate intraoperative image guidance.

Description

用于微创全膝关节置换手术的增强现实导航方法及系统Augmented reality navigation method and system for minimally invasive total knee replacement surgery
相关申请的交叉引用Cross references to related applications
本申请要求北京航天航空大学于2019年06月18日提交的、发明名称为“用于微创全膝关节置换手术的增强现实导航方法及系统”的、中国专利申请号“201910527900.4”的优先权。This application claims the priority of the Chinese patent application number “201910527900.4” submitted by Beijing University of Aeronautics and Astronautics on June 18, 2019, with the title of “Augmented Reality Navigation Method and System for Minimally Invasive Total Knee Replacement Surgery” .
技术领域Technical field
本发明涉及微创外科手术技术领域，特别涉及一种用于微创全膝关节置换手术的增强现实导航方法及系统。The invention relates to the technical field of minimally invasive surgery, in particular to an augmented reality navigation method and system for minimally invasive total knee replacement surgery.
背景技术Background technique
膝关节置换术，是将衬垫磨损的膝关节面切除，换上由金属、聚乙烯等材料制成的关节面，以达到解除患者疼痛、改善膝关节功能等目的。TKA(Total Knee Arthroplasty，全膝关节置换术)是目前治疗膝关节疾病的重要途径。膝关节结构复杂，手术空间狭小，周围存在重要的血管和神经，传统开放式手术易造成大出血及多种并发症。与之相反，MIS-TKA(Minimally Invasive Total Knee Arthroplasty，微创全膝关节置换术)以其创口小等优势而逐渐成为TKA手术的主流发展趋势。Knee replacement surgery is to remove the worn knee joint surface of the pad and replace it with a joint surface made of metal, polyethylene and other materials to relieve the patient's pain and improve the function of the knee joint. TKA (Total Knee Arthroplasty, total knee arthroplasty) is currently an important way to treat knee joint diseases. The structure of the knee joint is complex, the operation space is small, and there are important blood vessels and nerves around it. Traditional open surgery is likely to cause hemorrhage and many complications. In contrast, MIS-TKA (Minimally Invasive Total Knee Arthroplasty, minimally invasive total knee replacement) has gradually become the mainstream development trend of TKA surgery due to its advantages such as small wounds.
但是，微创全膝关节置换术由于术野狭小，对于医生的经验技巧要求较高，易造成置入假体对线偏离，进而导致磨损、偏心负荷等问题，影响患者术后行动并缩短假体寿命。目前骨科微创手术中常采用关节镜或CT等图像引导手段辅助医生完成手术，但对于MIS-TKA手术，都或多或少存在手术环境感知有限，定位不便或引入辐射等问题。增强现实导航可以为医生提供术中引导，有效解决手术视野狭小，难以获取患部定位信息的问题，同时避免了辐射伤害。However, minimally invasive total knee arthroplasty due to the narrow surgical field and high requirements for the doctor’s experience and skills can easily cause the implanted prosthesis to deviate from the alignment, which will lead to problems such as wear and eccentric load, affecting the patient’s postoperative actions and shortening the false Body life. At present, in orthopedic minimally invasive surgery, image-guided methods such as arthroscopy or CT are often used to assist doctors in completing the operation, but for MIS-TKA surgery, there are more or less problems such as limited perception of the operating environment, inconvenient positioning or introduction of radiation. Augmented reality navigation can provide doctors with intraoperative guidance, effectively solve the problem of narrow surgical field of view and difficulty in obtaining location information of the affected part, while avoiding radiation damage.
鉴于骨科手术的特殊性，目前对于骨科手术的增强现实导航的研究，一般采用基于光学透视的增强现实导航方法，但受限于相关硬件技术的发展，对基于光学透视的术中增强现实导航技术的研究仍处于起步阶段。目前最先进的基于光学透视的增强现实设备是微软的HoloLens混合现实眼镜，现有与手术应用相关的对于此类增强现实导航的研究几乎全部基于HoloLens。使用HoloLens进行术中增强现实导航，需要解决的核心问题是如何将虚拟场景空间、术中现实场景空间与术前图像空间统一起来，以便将来自患者术前CT/MRI扫描的虚拟解剖模型以准确的位置和姿态叠加到佩戴HoloLens的医生的视野下。In view of the particularity of orthopedic surgery, the current research on augmented reality navigation for orthopedic surgery generally adopts the augmented reality navigation method based on optical fluoroscopy. However, it is limited by the development of related hardware technology. The research is still in its infancy. At present, the most advanced augmented reality device based on optical perspective is Microsoft's HoloLens mixed reality glasses. The existing research related to surgical applications for this type of augmented reality navigation is almost all based on HoloLens. Using HoloLens for intraoperative augmented reality navigation, the core problem that needs to be solved is how to unify the virtual scene space, intraoperative reality scene space and preoperative image space, so that the virtual anatomical model from the patient’s preoperative CT/MRI scan can be accurately The position and posture of is superimposed under the field of vision of the doctor wearing HoloLens.
现有研究主要分为三类：其一是直接将虚拟模型显示在患者体外仅供医生参考，在实际临床应用中意义不大；其二是通过HoloLens提供的语音、手势等交互功能，在视野下手动调整虚拟模型的位姿直至与手术部位重合，但此方法操作不便，耗时较多，显示精度上也难以保证；其三是通过HoloLens上的网络摄像头获取图像，利用单目视觉及图像识别技术确定坐标系之间的关系，但由于网络摄像头获取的图像与佩戴HoloLens的人看见的虚实场景的相对位置并不相同，最终的增强现实效果虽然在网络摄像头获取的图像中是正确的，但是在HoloLens的佩戴者看来虚拟模型和实际患部位置存在一定的偏差。Existing research is mainly divided into three categories: one is to directly display the virtual model outside the patient's body for doctors' reference, which is of little significance in actual clinical applications; the other is to provide interactive functions such as voice and gestures through HoloLens, Manually adjust the pose of the virtual model until it coincides with the surgical site, but this method is inconvenient, time-consuming, and difficult to guarantee the display accuracy; the third is to obtain images through the webcam on the HoloLens, using monocular vision and images The recognition technology determines the relationship between the coordinate systems, but because the relative position of the image obtained by the webcam and the virtual and real scene seen by the person wearing HoloLens is not the same, the final augmented reality effect is correct in the image obtained by the webcam. However, in the eyes of the HoloLens wearer, there is a certain deviation between the virtual model and the actual position of the affected part.
发明内容Summary of the invention
本发明旨在至少在一定程度上解决相关技术中的技术问题之一。The present invention aims to solve one of the technical problems in the related art at least to a certain extent.
为此，本发明的一个目的在于提出一种用于微创全膝关节置换手术的增强现实导航方法。To this end, an object of the present invention is to provide an augmented reality navigation method for minimally invasive total knee replacement surgery.
本发明的另一个目的在于提出一种用于微创全膝关节置换手术的增强现实导航系统。Another object of the present invention is to provide an augmented reality navigation system for minimally invasive total knee replacement surgery.
为达到上述目的，本发明一方面实施例提出了一种用于微创全膝关节置换手术的增强现实导航方法，包括：步骤S1：获取HoloLens应用程序对应虚拟空间的世界坐标系与现实场景的坐标系之间的第一关系；步骤S2：将术中膝关节点云和术前通过CT扫描获得的三维模型点云按照空间变换匹配，得到术前医学图像空间坐标系和双目相机坐标系的第二关系；步骤S3：根据所述第一关系和所述第二关系将虚拟股骨、胫骨模型及对应的手术导板模型叠加到HoloLens视野下，实现增强现实导航。In order to achieve the above objective, an embodiment of the present invention proposes an augmented reality navigation method for minimally invasive total knee replacement surgery, including: Step S1: Obtain the world coordinate system and the real scene of the virtual space corresponding to the HoloLens application The first relationship between the coordinate systems; Step S2: Match the intraoperative knee joint point cloud with the three-dimensional model point cloud obtained by CT scanning before the operation according to the spatial transformation to obtain the preoperative medical image spatial coordinate system and the binocular camera coordinate system Step S3: According to the first relationship and the second relationship, the virtual femur, tibia model and the corresponding surgical guide model are superimposed on the HoloLens field of view to achieve augmented reality navigation.
本发明实施例的用于微创全膝关节置换手术的增强现实导航方法，可实现HoloLens虚拟空间坐标系的半自动化标定，并结合图像配准技术将虚拟膝关节解剖模型及虚拟手术导板模型准确叠加至对应真实患部位置，实现增强现实导航，从而可以为医生提供直观准确的术中图像引导。The augmented reality navigation method for minimally invasive total knee replacement surgery of the embodiment of the present invention can realize the semi-automatic calibration of the HoloLens virtual space coordinate system, and combine the image registration technology to accurately make the virtual knee joint anatomical model and virtual surgical guide model It is superimposed to the corresponding actual position of the affected part to realize augmented reality navigation, which can provide doctors with intuitive and accurate intraoperative image guidance.
另外，根据本发明上述实施例的用于微创全膝关节置换手术的增强现实导航方法还可以具有以下附加的技术特征：In addition, the augmented reality navigation method for minimally invasive total knee replacement surgery according to the above embodiment of the present invention may also have the following additional technical features:
进一步地，在本发明的一个实施例中，所述步骤S1包括：采用双目相机和视觉标记物辅助标定HoloLens虚拟场景空间坐标系。Further, in an embodiment of the present invention, the step S1 includes: using a binocular camera and a visual marker to assist in calibrating the HoloLens virtual scene space coordinate system.
进一步地，在本发明的一个实施例中，所述步骤S1进一步包括：固定所述双目相机，并将所述HoloLens置于所述双目相机的视野下；采集标记物坐标系相对于相机坐标系的位姿，并采集所述HoloLens应用程序的虚拟场景世界坐标系相对于HoloLens自身坐标系的位姿，以得到多组位姿数据；根据所述多组位姿数据得到所述双目相机坐标系相对于虚拟场景世界坐标系的位姿关系，以得到所述虚拟空间的世界坐标系与所述现实场景的坐标系 之间的第一关系。Further, in an embodiment of the present invention, the step S1 further includes: fixing the binocular camera, and placing the HoloLens under the field of view of the binocular camera; and collecting the marker coordinate system relative to the camera The pose of the coordinate system, and collect the pose of the virtual scene world coordinate system of the HoloLens application relative to the HoloLens own coordinate system to obtain multiple sets of pose data; obtain the binoculars according to the multiple sets of pose data The pose relationship between the camera coordinate system and the virtual scene world coordinate system to obtain the first relationship between the world coordinate system of the virtual space and the coordinate system of the real scene.
进一步地，在本发明的一个实施例中，所述步骤S2包括：将视觉探针和第一视觉标记物及第二视觉标记物辅助采集点云，其中，所述第一视觉标记物及所述第二视觉标记物被分别固定于股骨和胫骨上，以得到所述术中膝关节点云；采用随机采样一致配准算法结合迭代最近点算法完成配准。Further, in an embodiment of the present invention, the step S2 includes: assisting the collection of the point cloud by the visual probe, the first visual marker and the second visual marker, wherein the first visual marker and the second visual marker The second visual markers are respectively fixed on the femur and tibia to obtain the intraoperative knee joint point cloud; the random sampling uniform registration algorithm is combined with the iterative closest point algorithm to complete the registration.
进一步地，在本发明的一个实施例中，所述虚拟股骨在所述虚拟场景空间世界坐标系下的位姿的计算公式为：Further, in an embodiment of the present invention, the formula for calculating the pose of the virtual femur in the virtual scene space world coordinate system is:
Figure PCTCN2020079316-appb-000001
Figure PCTCN2020079316-appb-000001
其中，
Figure PCTCN2020079316-appb-000002
是双目相机坐标系相对于虚拟空间的世界坐标系的位姿关系，
Figure PCTCN2020079316-appb-000003
是股骨上的第一视觉标记物坐标系相对于双目相机坐标系的位姿，
Figure PCTCN2020079316-appb-000004
是CT坐标系相对于第一视觉标记物坐标系的位姿，P CT是所述虚拟股骨在CT坐标系下的位姿。 among them,
Figure PCTCN2020079316-appb-000002
Is the pose relationship between the binocular camera coordinate system and the world coordinate system of the virtual space,
Figure PCTCN2020079316-appb-000003
Is the pose of the first visual marker coordinate system on the femur relative to the coordinate system of the binocular camera,
Figure PCTCN2020079316-appb-000004
It is the pose of the CT coordinate system relative to the coordinate system of the first visual marker, and P CT is the pose of the virtual femur in the CT coordinate system.
为达到上述目的，本发明另一方面实施例提出了一种用于微创全膝关节置换手术的增强现实导航系统，包括：获取模块，用于获取HoloLens应用程序对应虚拟空间的世界坐标系与现实场景的坐标系之间的第一关系；匹配模块，用于将术中膝关节点云和术前通过CT扫描获得的三维模型点云按照空间变换匹配，得到术前医学图像空间坐标系和双目相机坐标系的第二关系；叠加模块，用于根据所述第一关系和所述第二关系将虚拟股骨、胫骨模型及对应的手术导板模型叠加到HoloLens视野下，实现增强现实导航。In order to achieve the above objective, another embodiment of the present invention proposes an augmented reality navigation system for minimally invasive total knee replacement surgery, including: an acquisition module for acquiring the world coordinate system and the virtual space corresponding to the HoloLens application The first relationship between the coordinate systems of the real scene; the matching module is used to match the intraoperative knee joint point cloud with the three-dimensional model point cloud obtained by CT scanning before the operation according to the spatial transformation to obtain the preoperative medical image spatial coordinate system and The second relationship of the binocular camera coordinate system; the superimposition module is used to superimpose the virtual femur, tibia model and the corresponding surgical guide model into the HoloLens field of view according to the first relationship and the second relationship to realize augmented reality navigation.
本发明实施例的用于微创全膝关节置换手术的增强现实导航系统，可实现HoloLens虚拟空间坐标系的半自动化标定，并结合图像配准技术将虚拟膝关节解剖模型及虚拟手术导板模型准确叠加至对应真实患部位置，实现增强现实导航，从而可以为医生提供直观准确的术中图像引导。The augmented reality navigation system for minimally invasive total knee replacement surgery of the embodiment of the present invention can realize the semi-automatic calibration of the HoloLens virtual space coordinate system, and combine the image registration technology to accurately make the virtual knee joint anatomical model and virtual surgical guide model It is superimposed to the corresponding actual position of the affected part to realize augmented reality navigation, which can provide doctors with intuitive and accurate intraoperative image guidance.
另外，根据本发明上述实施例的用于微创全膝关节置换手术的增强现实导航系统还可以具有以下附加的技术特征：In addition, the augmented reality navigation system for minimally invasive total knee replacement surgery according to the above embodiment of the present invention may also have the following additional technical features:
进一步地，在本发明的一个实施例中，所述获取模块进一步用于采用双目相机和视觉标记物辅助标定HoloLens虚拟场景空间坐标系。Further, in an embodiment of the present invention, the acquisition module is further configured to use a binocular camera and a visual marker to assist in calibrating the HoloLens virtual scene space coordinate system.
进一步地，在本发明的一个实施例中，所述获取模块进一步用于固定所述双目相机，并将所述HoloLens置于所述双目相机的视野下，并采集标记物坐标系相对于相机坐标系的位姿，并采集所述HoloLens应用程序的虚拟场景世界坐标系相对于HoloLens自身坐标系的位姿，以得到多组位姿数据，根据所述多组位姿数据得到所述双目相机坐标系相对于虚拟场景世界坐标系的位姿关系，以得到所述虚拟空间的世界坐标系与所述现实场景的坐标系之间的第一关系。Further, in an embodiment of the present invention, the acquisition module is further used to fix the binocular camera, place the HoloLens under the field of view of the binocular camera, and collect the marker coordinate system relative to The pose of the camera coordinate system, and the pose of the virtual scene world coordinate system of the HoloLens application relative to the HoloLens's own coordinate system is collected to obtain multiple sets of pose data, and the dual poses are obtained according to the multiple sets of pose data. The pose relationship of the camera coordinate system with respect to the world coordinate system of the virtual scene to obtain the first relationship between the world coordinate system of the virtual space and the coordinate system of the real scene.
进一步地，在本发明的一个实施例中，所述匹配模块进一步用于将视觉探针和第一视觉标记物及第二视觉标记物辅助采集点云，其中，所述第一视觉标记物及所述第二视觉标记物被分别固定于股骨和胫骨上，以得到所述术中膝关节点云，并采用随机采样一致配准算法结合迭代最近点算法完成配准。Further, in an embodiment of the present invention, the matching module is further used to assist in collecting the point cloud by the vision probe, the first visual marker and the second visual marker, wherein the first visual marker and The second visual markers are respectively fixed on the femur and tibia to obtain the intraoperative knee joint point cloud, and the random sampling consistent registration algorithm combined with the iterative closest point algorithm is used to complete the registration.
进一步地，在本发明的一个实施例中，所述虚拟股骨在所述虚拟场景空间世界坐标系下的位姿的计算公式为：Further, in an embodiment of the present invention, the formula for calculating the pose of the virtual femur in the virtual scene space world coordinate system is:
Figure PCTCN2020079316-appb-000005
Figure PCTCN2020079316-appb-000005
其中，
Figure PCTCN2020079316-appb-000006
是双目相机坐标系相对于虚拟空间的世界坐标系的位姿关系，
Figure PCTCN2020079316-appb-000007
是股骨上的第一视觉标记物坐标系相对于双目相机坐标系的位姿，
Figure PCTCN2020079316-appb-000008
是CT坐标系相对于第一视觉标记物坐标系的位姿，P CT是所述虚拟股骨在CT坐标系下的位姿。 among them,
Figure PCTCN2020079316-appb-000006
Is the pose relationship between the binocular camera coordinate system and the world coordinate system of the virtual space,
Figure PCTCN2020079316-appb-000007
Is the pose of the coordinate system of the first visual marker on the femur relative to the coordinate system of the binocular camera,
Figure PCTCN2020079316-appb-000008
Is the pose of the CT coordinate system relative to the coordinate system of the first visual marker, and P CT is the pose of the virtual femur in the CT coordinate system.
本发明附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本发明的实践了解到。The additional aspects and advantages of the present invention will be partly given in the following description, and partly will become obvious from the following description, or be understood through the practice of the present invention.
附图说明Description of the drawings
本发明上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：The above and/or additional aspects and advantages of the present invention will become obvious and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, in which:
图1为根据本发明一个实施例的用于微创全膝关节置换手术的增强现实导航方法的流程图；Fig. 1 is a flowchart of an augmented reality navigation method for minimally invasive total knee replacement surgery according to an embodiment of the present invention;
图2为根据本发明一个实施例的标记物被固定于HoloLens上的示意图；Figure 2 is a schematic diagram of a marker fixed on HoloLens according to an embodiment of the present invention;
图3为根据本发明一个实施例的各坐标系及坐标系间转化关系示意图；Fig. 3 is a schematic diagram of coordinate systems and transformation relationships between coordinate systems according to an embodiment of the present invention;
图4为根据本发明一个实施例的采集点云示意图；4 is a schematic diagram of collecting point clouds according to an embodiment of the present invention;
图5为根据本发明一个示例采集点云示意图；Fig. 5 is a schematic diagram of collecting point clouds according to an example of the present invention;
图6为根据本发明一个实施例的点云渲染示意图；Fig. 6 is a schematic diagram of point cloud rendering according to an embodiment of the present invention;
图7为根据本发明一个实施例的配准结果示意图；Fig. 7 is a schematic diagram of a registration result according to an embodiment of the present invention;
图8为根据本发明一个实施例的坐标系关系示意图；Fig. 8 is a schematic diagram of a coordinate system relationship according to an embodiment of the present invention;
图9为根据本发明一个实施例的增强现实显示效果示意图；FIG. 9 is a schematic diagram of an augmented reality display effect according to an embodiment of the present invention;
图10为根据本发明一个实施例的用于微创全膝关节置换手术的增强现实导航系统的结构示意图。Fig. 10 is a schematic structural diagram of an augmented reality navigation system for minimally invasive total knee replacement surgery according to an embodiment of the present invention.
具体实施方式Detailed ways
下面详细描述本发明的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本发明，而不能理解为对本发明的限制。The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention, but should not be construed as limiting the present invention.
下面参照附图描述根据本发明实施例提出的用于微创全膝关节置换手术的增强现实导航方法及系统，首先将参照附图描述根据本发明实施例提出的用于微创全膝关节置换手术的增强现实导航方法。The following describes the augmented reality navigation method and system for minimally invasive total knee replacement surgery proposed according to the embodiments of the present invention with reference to the accompanying drawings. First, the proposed method and system for minimally invasive total knee replacement surgery proposed by the present invention will be described with reference to the accompanying drawings. Augmented reality navigation method for surgery.
图1是本发明一个实施例的用于微创全膝关节置换手术的增强现实导航方法的流程图。Fig. 1 is a flowchart of an augmented reality navigation method for minimally invasive total knee replacement surgery according to an embodiment of the present invention.
如图1所示，该用于微创全膝关节置换手术的增强现实导航方法包括以下步骤：As shown in Figure 1, the augmented reality navigation method for minimally invasive total knee replacement surgery includes the following steps:
步骤S1：获取HoloLens应用程序对应虚拟空间的世界坐标系与现实场景的坐标系之间的第一关系。Step S1: Obtain the first relationship between the world coordinate system of the virtual space corresponding to the HoloLens application and the coordinate system of the real scene.
可以理解的是，步骤S1主要用于HoloLens标定，其中，本发明实施例可以通过一定的手段获得HoloLens应用程序对应虚拟空间的世界坐标系C HG与现实场景的坐标系C C之间的关系，从而实现HoloLens标定。需要说明的是，现实场景的坐标系C C实际由双目相机坐标系表征。 It is understandable that step S1 is mainly used for HoloLens calibration. The embodiment of the present invention can obtain the relationship between the world coordinate system C HG of the virtual space corresponding to the HoloLens application and the coordinate system C C of the real scene through certain means. So as to achieve HoloLens calibration. It should be noted that the coordinate system C C of the real scene is actually represented by the coordinate system of the binocular camera.
另外，HoloLens应用程序为用户开发并安装在HoloLens中的软件，类似于手机的APP(Application，应用程序)。在HoloLens应用程序启动后，自动创建HoloLens应用程序对应的虚拟空间的世界坐标系，并且该坐标系会一直存在直到用户关闭应用程序。本发明实施例还可以通过微软官方提供的程序接口显示一个虚拟模型，其中，虚拟模型可以在距离世界坐标系C HG原点一定距离的位置、且相对于坐标轴方向以一定姿态显示。 In addition, the HoloLens application is software developed for users and installed in HoloLens, which is similar to an APP (Application) of a mobile phone. After the HoloLens application is started, the world coordinate system of the virtual space corresponding to the HoloLens application is automatically created, and the coordinate system will always exist until the user closes the application. The embodiment of the present invention can also display a virtual model through a program interface officially provided by Microsoft, where the virtual model can be displayed at a certain distance from the origin of the world coordinate system C HG and in a certain posture relative to the coordinate axis direction.
举例而言，HoloLens应用程序置入一个立方体模型，并设定立方体模型的位置是(1m,1m,1m)，则打开应用程序之后，立方体模型在虚拟世界坐标系下的位置为(1m,1m,1m)；同样，可以在虚拟场景空间世界坐标系中设置立方体模型旋转姿态。由于往往并不知道虚拟场景空间世界坐标系与现实场景的坐标系之间的关系，也不知道在虚拟模型在现实场景中显示时会呈现何种姿态，因此，往往难以在准确的在现实场景中显示虚拟模型。而本发明实施例可以有效解决该问题，通过获取虚拟模型与现实中的物体(比如，双目相机)之间的坐标系关系，从而只需要获取在现实中哪个位置、何种姿态下显示虚拟模型，便可以将需要在现实显示虚拟模型的坐标信息转化到虚拟场景空间的世界坐标系下，进而以准确的位姿显示虚拟模型。For example, the HoloLens application inserts a cube model and sets the position of the cube model to (1m, 1m, 1m), then after opening the application, the position of the cube model in the virtual world coordinate system is (1m, 1m) ,1m); Similarly, the rotation posture of the cube model can be set in the virtual scene space world coordinate system. Because the relationship between the virtual scene space world coordinate system and the real scene coordinate system is often not known, and the posture of the virtual model when displayed in the real scene is not known, it is often difficult to accurately display in the real scene. The virtual model is displayed in. The embodiment of the present invention can effectively solve this problem. By obtaining the coordinate system relationship between the virtual model and the object in reality (for example, binocular camera), it is only necessary to obtain which position and posture in reality to display the virtual model. The model can transform the coordinate information of the virtual model that needs to be displayed in reality to the world coordinate system of the virtual scene space, and then display the virtual model in an accurate pose.
进一步地，在本发明的一个实施例中，步骤S1包括：采用双目相机和视觉标记物辅助标定HoloLens虚拟场景空间坐标系。Further, in an embodiment of the present invention, step S1 includes: using a binocular camera and a visual marker to assist in calibrating the HoloLens virtual scene space coordinate system.
可以理解的是，HoloLens除了包括虚拟空间世界坐标系外，还包括用于表征其自身位 姿的局部坐标系C HL，双目相机的位姿随着HoloLens的移动和旋转而改变，其中，HoloLens的移动和旋转可以被双目相机内部的传感器感知到。本发明实施例的方法可以采用双目相机和视觉标记物辅助标定HoloLens虚拟场景空间坐标系。如图2所示，标记物固定于HoloLens上，其中，标记物中含有几个黑白相间、且易于识别的X角点。 It is understandable that, in addition to the virtual space world coordinate system, HoloLens also includes the local coordinate system C HL used to characterize its own pose. The pose of the binocular camera changes with the movement and rotation of the HoloLens. HoloLens The movement and rotation of the camera can be sensed by the sensors inside the binocular camera. The method of the embodiment of the present invention may use binocular cameras and visual markers to assist in calibrating the HoloLens virtual scene spatial coordinate system. As shown in Figure 2, the marker is fixed on the HoloLens, where the marker contains several black and white X corners that are easy to identify.
进一步地，在本发明的一个实施例中，步骤S1进一步包括：固定双目相机，并将HoloLens置于双目相机的视野下；采集标记物坐标系相对于相机坐标系的位姿，并采集HoloLens应用程序的虚拟场景世界坐标系相对于HoloLens自身坐标系的位姿，以得到多组位姿数据；根据多组位姿数据得到双目相机坐标系相对于虚拟场景世界坐标系的位姿关系，以得到虚拟空间的世界坐标系与现实场景的坐标系之间的第一关系。Further, in an embodiment of the present invention, step S1 further includes: fixing the binocular camera and placing the HoloLens under the field of view of the binocular camera; collecting the pose of the marker coordinate system relative to the camera coordinate system, and collecting The pose of the virtual scene world coordinate system of the HoloLens application relative to the HoloLens own coordinate system to obtain multiple sets of pose data; according to the multiple sets of pose data, the pose relationship between the binocular camera coordinate system and the virtual scene world coordinate system is obtained , In order to obtain the first relationship between the world coordinate system of the virtual space and the coordinate system of the real scene.
可以理解的是，本发明实施例可以通过双目相机获取图片并识别图片中的X角点，可以根据双目视觉计算出角点在双目相机坐标系中的三维坐标信息(这些可以是在个人计算机中完成的，计算机与双目相机连接，实时处理相机抓取的图片)，进而计算出标记物坐标系(记为C HM)相对于相机坐标系的位姿，该相对位姿记为
Figure PCTCN2020079316-appb-000009
其中，标记物坐标系可以由4个角点定义，角点连成的四边形的重心为标记物坐标系原点，当然也可以采用别的方式定义，在此不做具体限定。 It is understandable that the embodiment of the present invention can obtain a picture through a binocular camera and recognize the X corner point in the picture, and can calculate the three-dimensional coordinate information of the corner point in the binocular camera coordinate system according to the binocular vision (these can be in Completed in the personal computer, the computer is connected with the binocular camera, and the pictures captured by the camera are processed in real time), and then the position of the marker coordinate system (denoted as C HM ) relative to the camera coordinate system is calculated, and the relative position is denoted as
Figure PCTCN2020079316-appb-000009
Among them, the marker coordinate system can be defined by four corner points, and the center of gravity of the quadrilateral formed by the corner points is the origin of the marker coordinate system. Of course, it can also be defined in other ways, which is not specifically limited here.
具体而言，标定步骤如下：首先，固定双目相机，将HoloLens置于双目相机视野下，保证上面的标记物同时处于两个镜头的视野之内。采集此时标记物坐标系相对于相机坐标系的位姿。与此同时，通过个人计算机和HoloLens之间的无线网络通信，采集HoloLens应用程序的虚拟场景世界坐标系相对于HoloLens自身坐标系的位姿，记为
Figure PCTCN2020079316-appb-000010
改变HoloLens的位置和姿态，重复以上步骤，采集多组位姿数据。最终需要得到的是双目相机坐标系相对于虚拟场景世界坐标系的位姿关系，记为
Figure PCTCN2020079316-appb-000011
相关的位姿关系展示在图3中，并可以用等式描述为： Specifically, the calibration steps are as follows: First, fix the binocular camera and place the HoloLens under the field of view of the binocular camera to ensure that the markers on it are in the field of view of the two lenses at the same time. Collect the pose of the marker coordinate system relative to the camera coordinate system at this time. At the same time, through the wireless network communication between the personal computer and HoloLens, the position and pose of the virtual scene world coordinate system of the HoloLens application relative to the HoloLens's own coordinate system is collected, which is recorded as
Figure PCTCN2020079316-appb-000010
Change the position and posture of HoloLens, repeat the above steps to collect multiple sets of posture data. What needs to be finally obtained is the pose relationship between the binocular camera coordinate system and the virtual scene world coordinate system, denoted as
Figure PCTCN2020079316-appb-000011
The related pose relationship is shown in Figure 3, and can be described as:
Figure PCTCN2020079316-appb-000012
Figure PCTCN2020079316-appb-000012
其中，
Figure PCTCN2020079316-appb-000013
和
Figure PCTCN2020079316-appb-000014
分别是第i组和第j组中的
Figure PCTCN2020079316-appb-000015
和
Figure PCTCN2020079316-appb-000016
同理。采用类似机器人手眼标定的方法可以求得
Figure PCTCN2020079316-appb-000017
的最小二乘解。 among them,
Figure PCTCN2020079316-appb-000013
with
Figure PCTCN2020079316-appb-000014
Are the i-th group and j-th group
Figure PCTCN2020079316-appb-000015
with
Figure PCTCN2020079316-appb-000016
Similarly. It can be obtained by a method similar to robot hand-eye calibration
Figure PCTCN2020079316-appb-000017
The least squares solution of.
步骤S2：将术中膝关节点云和术前通过CT扫描获得的三维模型点云按照空间变换匹配，得到术前医学图像空间坐标系和双目相机坐标系的第二关系。Step S2: Match the intraoperative knee joint point cloud and the three-dimensional model point cloud obtained through the CT scan before the operation according to the spatial transformation to obtain the second relationship between the preoperative medical image spatial coordinate system and the binocular camera coordinate system.
可以理解的是，步骤S2主要用于膝关节表面点云配准，增强现实导航所使用的虚拟模型来自于术前CT/MRI扫描，为了准确显示虚拟膝关节模型，还需要通过图像配准获得术 前医学图像空间坐标系C CT和双目相机坐标系C Cp的关系。其中，配准是指将术中膝关节点云和术前扫描并处理获得的三维模型点云按照一定的空间变换匹配起来。 It is understandable that step S2 is mainly used for knee joint surface point cloud registration. The virtual model used in augmented reality navigation comes from preoperative CT/MRI scans. In order to accurately display the virtual knee joint model, it needs to be obtained through image registration. The relationship between the preoperative medical image spatial coordinate system C CT and the binocular camera coordinate system C C p. Among them, registration refers to matching the intraoperative knee joint point cloud with the 3D model point cloud obtained by scanning and processing before the operation according to a certain spatial transformation.
进一步地，在本发明的一个实施例中，步骤S2包括：将视觉探针和第一视觉标记物及第二视觉标记物辅助采集点云，其中，第一视觉标记物及第二视觉标记物被分别固定于股骨和胫骨上，以得到术中膝关节点云；采用随机采样一致配准算法结合迭代最近点算法完成配准。Further, in an embodiment of the present invention, step S2 includes: assisting the collection of the point cloud by the vision probe, the first visual marker and the second visual marker, wherein the first visual marker and the second visual marker They are respectively fixed on the femur and tibia to obtain the intraoperative knee joint point cloud; the random sampling consistent registration algorithm combined with the iterative closest point algorithm is used to complete the registration.
具体而言，(1)术中膝关节表面点云获取Specifically, (1) Intraoperative knee joint surface point cloud acquisition
全膝关节置换是对股骨和胫骨分别进行切骨和置入假体的手术，因此对于股骨和胫骨需要分别做配准。本发明实施例的方法采用一个视觉探针和两个视觉标记物辅助采集点云，如图4所示，两个标记物被分别固定于股骨和胫骨上。Total knee replacement is the operation of cutting the femur and tibia and inserting the prosthesis. Therefore, the femur and tibia need to be registered separately. The method of the embodiment of the present invention uses a vision probe and two vision markers to assist in collecting the point cloud. As shown in FIG. 4, the two markers are fixed on the femur and the tibia respectively.
以股骨为例，结合图5说明采集表面点云的过程：通过预先的注册，可以在双目相机识别到探针后计算出探针针尖点的三维坐标信息。在双目相机视野下，使用探针针尖贴在靠近关节部分的股骨表面上划动，计算机不断计算并记录每一帧相机抓取的图片中探针针尖点(也即股骨表面上的点)在股骨上标记物坐标系下的三维坐标信息。当探针在预先计划的区域划动完毕后即获得了所需的点云信息。在计算机程序中使用OpenGL渲染的点云如图6所示。Taking the femur as an example, the process of collecting surface point clouds is explained in conjunction with Figure 5: through pre-registration, the three-dimensional coordinate information of the probe tip point can be calculated after the binocular camera recognizes the probe. In the field of view of the binocular camera, use the probe tip to slide on the surface of the femur near the joint. The computer continuously calculates and records the point of the probe tip (that is, the point on the surface of the femur) in the pictures captured by the camera in each frame The three-dimensional coordinate information under the coordinate system of the marker on the femur. The required point cloud information is obtained after the probe has been swiped in the pre-planned area. The point cloud rendered using OpenGL in the computer program is shown in Figure 6.
(2)点云配准(2) Point cloud registration
由于股骨是刚性的，因此采用刚性配准算法即可。SAC-IA(Sample Consensus Initial Alignment，采用随机采样一致配准算法)结合传统ICP(Iterative Closest Point，迭代最近点算法)完成配准。Since the femur is rigid, the rigid registration algorithm can be used. SAC-IA (Sample Consensus Initial Alignment, using random sampling uniform registration algorithm) combined with traditional ICP (Iterative Closest Point, iterative closest point algorithm) to complete the registration.
需要说明的是，在刚性配准中，最经典的算法就是ICP算法，但是ICP依赖于良好的初始位姿估计，也就是说，需要给ICP算法一个良好的输入才能实现刚性配准，比如，在初始位姿估计时，需要初始时两个点云的位姿非常接近，如果初始时两个点云的位姿不是非常接近，则容易使得ICP算法陷入局部最优，从而导致非常坏的结果，进而难以完成刚性配准。因此，往往不直接使用ICP算法，而是使用ICP算法的变种，或者ICP结合其他算法一起使用，本领域技术人员可以根据实际情况选择，在此不做具体限定。为了实现刚性配准，本发明实施例通过SAC-IA进行粗配准，由于粗配准之后可以得到一个比较好的初始位姿估计，因此，本发明实施例再通过ICP算法即可以实现刚性配准。It should be noted that in rigid registration, the most classic algorithm is the ICP algorithm, but ICP relies on a good initial pose estimation, that is to say, a good input to the ICP algorithm is needed to achieve rigid registration, for example, In the initial pose estimation, the initial poses of the two point clouds need to be very close. If the initial poses of the two point clouds are not very close, it is easy to make the ICP algorithm fall into the local optimum, which will lead to very bad results. , And then it is difficult to complete rigid registration. Therefore, the ICP algorithm is often not used directly, but a variant of the ICP algorithm, or the ICP is used in combination with other algorithms. Those skilled in the art can choose according to the actual situation, and there is no specific limitation here. In order to achieve rigid registration, the embodiment of the present invention uses SAC-IA to perform coarse registration. Since a relatively good initial pose estimation can be obtained after the coarse registration, the embodiment of the present invention can realize rigid registration through the ICP algorithm. quasi.
由于术中划取的点云规模比较小，本发明实施例以划取的点云作为源点云，术前获得的点云作为目标点云，即，对术中获取的点云做空间变换，将点云变换到与术前CT扫描获得的点云近乎一致的位置。SAC-IA算法首先提取源点云中点的三维法线信息，并使用FPFH(Fast Point Feature Histogram，快速点特征直方图)特征，然后对目标点云做相同的 处理。通过在目标点云中寻找与在源点云中选取的一些点的FPFH特征近似的点，获得匹配的点对，并计算出点对间的最小二乘变换作为配准结果。然后，将此结果作为ICP算法的初始位姿估计，使用ICP算法继续迭代至收敛，获得的结果即为最终的配准变换结果。配准结果示意图7所示，其中，白色为术中划取的点云，黑色为患者术前CT扫描后经过图像分割等处理得到的三维模型，左边为股骨，右边为胫骨。Since the scale of the point cloud drawn during the operation is relatively small, the embodiment of the present invention uses the drawn point cloud as the source point cloud, and the point cloud obtained before the operation as the target point cloud, that is, spatial transformation is performed on the point cloud obtained during the operation. , Transform the point cloud to a position that is almost the same as the point cloud obtained by the preoperative CT scan. The SAC-IA algorithm first extracts the three-dimensional normal information of the points in the source point cloud, and uses FPFH (Fast Point Feature Histogram) features, and then performs the same processing on the target point cloud. By finding points in the target point cloud that are similar to the FPFH features of some points selected in the source point cloud, matching point pairs are obtained, and the least square transformation between the point pairs is calculated as the registration result. Then, use this result as the initial pose estimation of the ICP algorithm, and use the ICP algorithm to continue iterating until convergence, and the result obtained is the final registration transformation result. The registration result is shown in schematic diagram 7, where white is the point cloud drawn during the operation, and black is the three-dimensional model obtained by image segmentation after the preoperative CT scan of the patient. The femur is on the left and the tibia is on the right.
步骤S3：根据第一关系和第二关系将虚拟股骨、胫骨模型及对应的手术导板模型叠加到HoloLens视野下，实现增强现实导航。Step S3: According to the first relationship and the second relationship, the virtual femur, tibia model and the corresponding surgical guide model are superimposed on the HoloLens field of view to realize augmented reality navigation.
可以理解的是，步骤S3主要用于增强现实显示，在获得坐标系之间的关系后，虚拟股骨，胫骨模型及它们对应的手术导板模型就可以叠加到HoloLens视野下。仍然以股骨为例说明：记股骨上标记物坐标系为C FM。记最终配准结果，即C FM相对于C CT的位姿为
Figure PCTCN2020079316-appb-000018
相关的坐标系关系如图8所示。 It is understandable that step S3 is mainly used for augmented reality display. After obtaining the relationship between the coordinate systems, the virtual femur, tibia model and their corresponding surgical guide model can be superimposed under the HoloLens field of view. Still take the femur as an example: remember the coordinate system of the marker on the femur as C FM . Record the final registration result, that is, the pose of C FM relative to C CT is
Figure PCTCN2020079316-appb-000018
The related coordinate system relationship is shown in Figure 8.
那么虚拟股骨模型在虚拟场景空间世界坐标系下的位姿表示P HG可通过下式计算： Then the pose representation P HG of the virtual femur model in the virtual scene space world coordinate system can be calculated by the following formula:
Figure PCTCN2020079316-appb-000019
Figure PCTCN2020079316-appb-000019
其中，
Figure PCTCN2020079316-appb-000020
是双目相机坐标系相对于虚拟空间的世界坐标系的位姿关系，
Figure PCTCN2020079316-appb-000021
是股骨上的第一视觉标记物坐标系相对于双目相机坐标系的位姿，
Figure PCTCN2020079316-appb-000022
是上述配准结果
Figure PCTCN2020079316-appb-000023
的逆，表示CT坐标系相对于第一视觉标记物坐标系的位姿，P CT是虚拟股骨模型在CT坐标系下的位姿，若以矩阵表示则为单位矩阵。将计算出的位姿通过无线网络通信从个人计算机发送给HoloLens，就可以利用这个位姿信息完成最终的增强现实显示效果，效果示意图如9所示。 among them,
Figure PCTCN2020079316-appb-000020
Is the pose relationship between the binocular camera coordinate system and the world coordinate system of the virtual space,
Figure PCTCN2020079316-appb-000021
Is the pose of the coordinate system of the first visual marker on the femur relative to the coordinate system of the binocular camera,
Figure PCTCN2020079316-appb-000022
Is the result of the above registration
Figure PCTCN2020079316-appb-000023
The inverse of represents the pose of the CT coordinate system relative to the coordinate system of the first visual marker. P CT is the pose of the virtual femur model in the CT coordinate system. If expressed in a matrix, it is the identity matrix. Send the calculated pose from a personal computer to HoloLens through wireless network communication, and then you can use this pose information to complete the final augmented reality display effect. The effect diagram is shown in 9.
根据本发明实施例提出的用于微创全膝关节置换手术的增强现实导航方法，可实现HoloLens虚拟空间坐标系的半自动化标定，并结合图像配准技术将虚拟膝关节解剖模型及虚拟手术导板模型准确叠加至对应真实患部位置，实现增强现实导航，从而可以为医生提供直观准确的术中图像引导。The augmented reality navigation method for minimally invasive total knee replacement surgery proposed according to the embodiment of the present invention can realize the semi-automatic calibration of the HoloLens virtual space coordinate system, and combine the image registration technology to combine the virtual knee joint anatomical model and virtual surgical guide The model is accurately superimposed on the actual location of the affected part to achieve augmented reality navigation, which can provide doctors with intuitive and accurate intraoperative image guidance.
其次参照附图描述根据本发明实施例提出的用于微创全膝关节置换手术的增强现实导航系统。Next, an augmented reality navigation system for minimally invasive total knee replacement surgery proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.
图10是本发明一个实施例的用于微创全膝关节置换手术的增强现实导航系统的结构示意图。Fig. 10 is a schematic structural diagram of an augmented reality navigation system for minimally invasive total knee replacement surgery according to an embodiment of the present invention.
如图10所示，该用于微创全膝关节置换手术的增强现实导航系统10包括：获取模块100、匹配模块200和叠加模块300。As shown in FIG. 10, the augmented reality navigation system 10 for minimally invasive total knee replacement surgery includes: an acquisition module 100, a matching module 200 and an overlay module 300.
其中，获取模块100用于获取HoloLens应用程序对应虚拟空间的世界坐标系与现实场 景的坐标系之间的第一关系。匹配模块200用于将术中膝关节点云和术前通过CT扫描获得的三维模型点云按照空间变换匹配，得到术前医学图像空间坐标系和双目相机坐标系的第二关系。叠加模块300用于根据第一关系和第二关系将虚拟股骨、胫骨模型及对应的手术导板模型叠加到HoloLens视野下，实现增强现实导航。本发明实施例的系统10可实现HoloLens虚拟空间坐标系的半自动化标定，并结合图像配准技术将虚拟膝关节解剖模型及虚拟手术导板模型准确叠加至对应真实患部位置，从而可以为医生提供直观准确的术中图像引导。The acquiring module 100 is used to acquire the first relationship between the world coordinate system of the virtual space corresponding to the HoloLens application and the coordinate system of the real scene. The matching module 200 is configured to match the intraoperative knee joint point cloud with the three-dimensional model point cloud obtained through the CT scan before the operation according to the spatial transformation to obtain the second relationship between the preoperative medical image spatial coordinate system and the binocular camera coordinate system. The superimposing module 300 is used to superimpose the virtual femur, tibia model and the corresponding surgical guide model into the HoloLens field of view according to the first relationship and the second relationship, so as to realize augmented reality navigation. The system 10 of the embodiment of the present invention can realize the semi-automatic calibration of the HoloLens virtual space coordinate system, and combine the image registration technology to accurately superimpose the virtual knee joint anatomical model and the virtual surgical guide model to the corresponding real affected position, so as to provide the doctor with intuitive Accurate intraoperative image guidance.
进一步地，在本发明的一个实施例中，获取模块100进一步用于采用双目相机和视觉标记物辅助标定HoloLens虚拟场景空间坐标系。Further, in an embodiment of the present invention, the acquisition module 100 is further configured to use binocular cameras and visual markers to assist in calibrating the HoloLens virtual scene space coordinate system.
进一步地，在本发明的一个实施例中，获取模块100进一步用于固定双目相机，并将HoloLens置于双目相机的视野下，并采集标记物坐标系相对于相机坐标系的位姿，并采集HoloLens应用程序的虚拟场景世界坐标系相对于HoloLens自身坐标系的位姿，以得到多组位姿数据，根据多组位姿数据得到双目相机坐标系相对于虚拟场景世界坐标系的位姿关系，以得到虚拟空间的世界坐标系与现实场景的坐标系之间的第一关系。Further, in an embodiment of the present invention, the acquisition module 100 is further used to fix the binocular camera, place the HoloLens under the field of view of the binocular camera, and collect the pose of the marker coordinate system relative to the camera coordinate system, And collect the position and pose of the virtual scene world coordinate system of the HoloLens application relative to the HoloLens own coordinate system to obtain multiple sets of pose data. According to the multiple sets of pose data, obtain the position of the binocular camera coordinate system relative to the virtual scene world coordinate system. Posture relationship to obtain the first relationship between the world coordinate system of the virtual space and the coordinate system of the real scene.
进一步地，在本发明的一个实施例中，匹配模块200进一步用于将视觉探针和第一视觉标记物及第二视觉标记物辅助采集点云，其中，第一视觉标记物及第二视觉标记物被分别固定于股骨和胫骨上，以得到术中膝关节点云，并采用随机采样一致配准算法结合迭代最近点算法完成配准。Further, in an embodiment of the present invention, the matching module 200 is further used to assist the collection of the point cloud by the vision probe, the first vision marker and the second vision marker, wherein the first vision marker and the second vision marker The markers were fixed on the femur and tibia respectively to obtain the intraoperative knee joint point cloud, and the random sampling uniform registration algorithm combined with the iterative closest point algorithm was used to complete the registration.
进一步地，在本发明的一个实施例中，虚拟股骨在虚拟场景空间世界坐标系下的位姿的计算公式为：Further, in an embodiment of the present invention, the calculation formula of the pose of the virtual femur in the virtual scene space world coordinate system is:
Figure PCTCN2020079316-appb-000024
Figure PCTCN2020079316-appb-000024
其中，
Figure PCTCN2020079316-appb-000025
是双目相机坐标系相对于虚拟空间的世界坐标系的位姿关系，
Figure PCTCN2020079316-appb-000026
是股骨上的第一视觉标记物坐标系相对于双目相机坐标系的位姿，
Figure PCTCN2020079316-appb-000027
是CT坐标系相对于第一视觉标记物坐标系的位姿，P CT是所述虚拟股骨在CT坐标系下的位姿。 among them,
Figure PCTCN2020079316-appb-000025
Is the pose relationship between the binocular camera coordinate system and the world coordinate system of the virtual space,
Figure PCTCN2020079316-appb-000026
Is the pose of the coordinate system of the first visual marker on the femur relative to the coordinate system of the binocular camera,
Figure PCTCN2020079316-appb-000027
Is the pose of the CT coordinate system relative to the coordinate system of the first visual marker, and P CT is the pose of the virtual femur in the CT coordinate system.
需要说明的是，前述对用于微创全膝关节置换手术的增强现实导航方法实施例的解释说明也适用于该实施例的用于微创全膝关节置换手术的增强现实导航系统，此处不再赘述。It should be noted that the foregoing explanation of the embodiment of the augmented reality navigation method for minimally invasive total knee replacement surgery is also applicable to the augmented reality navigation system for minimally invasive total knee replacement surgery of this embodiment, here No longer.
根据本发明实施例的用于微创全膝关节置换手术的增强现实导航系统，可实现HoloLens虚拟空间坐标系的半自动化标定，并结合图像配准技术将虚拟膝关节解剖模型及虚拟手术导板模型准确叠加至对应真实患部位置，实现增强现实导航，从而可以为医生提供直观准确的术中图像引导。The augmented reality navigation system for minimally invasive total knee replacement surgery according to an embodiment of the present invention can realize the semi-automatic calibration of the HoloLens virtual space coordinate system, and combines the virtual knee joint anatomical model and virtual surgical guide model with image registration technology Accurately superimpose to the actual position of the affected part to achieve augmented reality navigation, which can provide doctors with intuitive and accurate intraoperative image guidance.
在本发明的描述中，需要理解的是，术语“中心”、“纵向”、“横向”、“长度”、“宽度”、 “厚度”、“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”“内”、“外”、“顺时针”、“逆时针”、“轴向”、“径向”、“周向”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本发明的限制。In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", " "Back", "Left", "Right", "Vertical", "Horizontal", "Top", "Bottom", "Inner", "Outer", "Clockwise", "Counterclockwise", "Axial", The orientation or positional relationship indicated by "radial", "circumferential", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying the pointed device or element It must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention.
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本发明的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.
在本发明中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”、“固定”等术语应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或成一体；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通或两个元件的相互作用关系，除非另有明确的限定。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本发明中的具体含义。In the present invention, unless otherwise clearly defined and defined, the terms "installed", "connected", "connected", "fixed" and other terms should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. , Or integrated; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediary, it can be the internal communication of two components or the interaction relationship between two components, unless otherwise specified The limit. For those of ordinary skill in the art, the specific meaning of the above-mentioned terms in the present invention can be understood according to specific circumstances.
在本发明中，除非另有明确的规定和限定，第一特征在第二特征“上”或“下”可以是第一和第二特征直接接触，或第一和第二特征通过中间媒介间接接触。而且，第一特征在第二特征“之上”、“上方”和“上面”可是第一特征在第二特征正上方或斜上方，或仅仅表示第一特征水平高度高于第二特征。第一特征在第二特征“之下”、“下方”和“下面”可以是第一特征在第二特征正下方或斜下方，或仅仅表示第一特征水平高度小于第二特征。In the present invention, unless expressly stipulated and defined otherwise, the first feature “on” or “under” the second feature may be in direct contact with the first and second features, or the first and second features may be indirectly through an intermediary. contact. Moreover, the "above", "above" and "above" of the first feature on the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the level of the first feature is higher than the second feature. The “below”, “below” and “below” of the second feature of the first feature may mean that the first feature is directly below or obliquely below the second feature, or it simply means that the level of the first feature is smaller than the second feature.
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean specific features described in conjunction with the embodiment or example , Structure, materials or features are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples and the features of the different embodiments or examples described in this specification without contradicting each other.
尽管上面已经示出和描述了本发明的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本发明的限制，本领域的普通技术人员在本发明的范围内可以对上述实施例进行变化、修改、替换和变型。Although the embodiments of the present invention have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can comment on the foregoing within the scope of the present invention. The embodiment undergoes changes, modifications, substitutions and modifications.

Claims (10)

  1. 一种用于微创全膝关节置换手术的增强现实导航方法，其特征在于，包括：An augmented reality navigation method for minimally invasive total knee replacement surgery, which is characterized in that it includes:
    步骤S1：获取HoloLens应用程序对应虚拟空间的世界坐标系与现实场景的坐标系之间的第一关系；Step S1: Obtain the first relationship between the world coordinate system of the virtual space corresponding to the HoloLens application and the coordinate system of the real scene;
    步骤S2：将术中膝关节点云和术前通过CT扫描获得的三维模型点云按照空间变换匹配，得到术前医学图像空间坐标系和双目相机坐标系的第二关系；以及Step S2: Match the intraoperative knee joint point cloud with the three-dimensional model point cloud obtained by CT scanning before the operation according to the spatial transformation, and obtain the second relationship between the preoperative medical image spatial coordinate system and the binocular camera coordinate system; and
    步骤S3：根据所述第一关系和所述第二关系将虚拟股骨、胫骨模型及对应的手术导板模型叠加到HoloLens视野下，实现增强现实导航。Step S3: According to the first relationship and the second relationship, the virtual femur, tibia model and the corresponding surgical guide model are superimposed on the HoloLens field of view to realize augmented reality navigation.
  2. 根据权利要求1所述的方法，其特征在于，所述步骤S1包括：The method according to claim 1, wherein the step S1 comprises:
    采用双目相机和视觉标记物辅助标定HoloLens虚拟场景空间坐标系。A binocular camera and visual markers are used to assist in the calibration of the HoloLens virtual scene space coordinate system.
  3. 根据权利要求2所述的方法，其特征在于，所述步骤S1进一步包括：The method according to claim 2, wherein the step S1 further comprises:
    固定所述双目相机，并将所述HoloLens置于所述双目相机的视野下；Fixing the binocular camera, and placing the HoloLens under the field of view of the binocular camera;
    采集标记物坐标系相对于相机坐标系的位姿，并采集所述HoloLens应用程序的虚拟场景世界坐标系相对于HoloLens自身坐标系的位姿，以得到多组位姿数据；Collecting the pose of the marker coordinate system relative to the camera coordinate system, and collecting the pose of the virtual scene world coordinate system of the HoloLens application relative to the HoloLens own coordinate system to obtain multiple sets of pose data;
    根据所述多组位姿数据得到所述双目相机坐标系相对于虚拟场景世界坐标系的位姿关系，以得到所述虚拟空间的世界坐标系与所述现实场景的坐标系之间的第一关系。Obtain the pose relationship between the binocular camera coordinate system and the virtual scene world coordinate system according to the multiple sets of pose data, so as to obtain the first position between the world coordinate system of the virtual space and the coordinate system of the real scene One relationship.
  4. 根据权利要求1所述的方法，其特征在于，所述步骤S2包括：The method according to claim 1, wherein the step S2 comprises:
    将视觉探针和第一视觉标记物及第二视觉标记物辅助采集点云，其中，所述第一视觉标记物及所述第二视觉标记物被分别固定于股骨和胫骨上，以得到所述术中膝关节点云；A visual probe and a first visual marker and a second visual marker are used to assist in collecting the point cloud, wherein the first visual marker and the second visual marker are respectively fixed on the femur and tibia to obtain the Describe the intraoperative knee joint point cloud;
    采用随机采样一致配准算法结合迭代最近点算法完成配准。The random sampling uniform registration algorithm combined with the iterative closest point algorithm is used to complete the registration.
  5. 根据权利要求1所述的方法，其特征在于，所述虚拟股骨在所述虚拟场景空间世界坐标系下的位姿的计算公式为：The method according to claim 1, wherein the formula for calculating the pose of the virtual femur in the world coordinate system of the virtual scene space is:
    Figure PCTCN2020079316-appb-100001
    Figure PCTCN2020079316-appb-100001
    其中，
    Figure PCTCN2020079316-appb-100002
    是双目相机坐标系相对于虚拟空间的世界坐标系的位姿关系，
    Figure PCTCN2020079316-appb-100003
    是股骨上的第一视觉标记物坐标系相对于双目相机坐标系的位姿，
    Figure PCTCN2020079316-appb-100004
    是CT坐标系相对于第一视觉标记物坐标系的位姿，P CT是所述虚拟股骨在CT坐标系下的位姿。     among them,
    Figure PCTCN2020079316-appb-100002
    Is the pose relationship between the binocular camera coordinate system and the world coordinate system of the virtual space,
    Figure PCTCN2020079316-appb-100003
    Is the pose of the coordinate system of the first visual marker on the femur relative to the coordinate system of the binocular camera,
    Figure PCTCN2020079316-appb-100004
    Is the pose of the CT coordinate system relative to the coordinate system of the first visual marker, and P CT is the pose of the virtual femur in the CT coordinate system.
  6. 一种用于微创全膝关节置换手术的增强现实导航系统，其特征在于，包括：An augmented reality navigation system for minimally invasive total knee replacement surgery, which is characterized in that it includes:
    获取模块，用于获取HoloLens应用程序对应虚拟空间的世界坐标系与现实场景的坐标系之间的第一关系；The obtaining module is used to obtain the first relationship between the world coordinate system of the virtual space corresponding to the HoloLens application and the coordinate system of the real scene;
    匹配模块，用于将术中膝关节点云和术前通过CT扫描获得的三维模型点云按照空间变 换匹配，得到术前医学图像空间坐标系和双目相机坐标系的第二关系；以及The matching module is used to match the knee joint point cloud during the operation with the 3D model point cloud obtained through CT scanning before the operation according to the spatial transformation to obtain the second relationship between the preoperative medical image spatial coordinate system and the binocular camera coordinate system; and
    叠加模块，用于根据所述第一关系和所述第二关系将虚拟股骨、胫骨模型及对应的手术导板模型叠加到HoloLens视野下，实现增强现实导航。The superimposing module is used to superimpose the virtual femur, tibia model and the corresponding surgical guide model into the HoloLens field of view according to the first relationship and the second relationship, so as to realize augmented reality navigation.
  7. 根据权利要求6所述的系统，其特征在于，所述获取模块进一步用于采用双目相机和视觉标记物辅助标定HoloLens虚拟场景空间坐标系。The system according to claim 6, wherein the acquisition module is further configured to use binocular cameras and visual markers to assist in calibrating the HoloLens virtual scene space coordinate system.
  8. 根据权利要求7所述的系统，其特征在于，所述获取模块进一步用于固定所述双目相机，并将所述HoloLens置于所述双目相机的视野下，并采集标记物坐标系相对于相机坐标系的位姿，并采集所述HoloLens应用程序的虚拟场景世界坐标系相对于HoloLens自身坐标系的位姿，以得到多组位姿数据，根据所述多组位姿数据得到所述双目相机坐标系相对于虚拟场景世界坐标系的位姿关系，以得到所述虚拟空间的世界坐标系与所述现实场景的坐标系之间的第一关系。The system according to claim 7, wherein the acquisition module is further configured to fix the binocular camera, place the HoloLens under the field of view of the binocular camera, and collect the relative coordinate system of the marker The pose in the camera coordinate system, and collect the pose of the virtual scene world coordinate system of the HoloLens application relative to the HoloLens own coordinate system to obtain multiple sets of pose data, and obtain the multiple sets of pose data. The pose relationship of the binocular camera coordinate system with respect to the world coordinate system of the virtual scene to obtain the first relationship between the world coordinate system of the virtual space and the coordinate system of the real scene.
  9. 根据权利要求6所述的系统，其特征在于，所述匹配模块进一步用于将视觉探针和第一视觉标记物及第二视觉标记物辅助采集点云，其中，所述第一视觉标记物及所述第二视觉标记物被分别固定于股骨和胫骨上，以得到所述术中膝关节点云，并采用随机采样一致配准算法结合迭代最近点算法完成配准。The system according to claim 6, wherein the matching module is further configured to assist in collecting the point cloud by the vision probe, the first visual marker and the second visual marker, wherein the first visual marker And the second visual markers are respectively fixed on the femur and tibia to obtain the intraoperative knee joint point cloud, and the random sampling consistent registration algorithm combined with the iterative closest point algorithm is used to complete the registration.
  10. 根据权利要求6所述的系统，其特征在于，所述虚拟股骨在所述虚拟场景空间世界坐标系下的位姿的计算公式为：The system according to claim 6, wherein the formula for calculating the pose of the virtual femur in the world coordinate system of the virtual scene space is:
    Figure PCTCN2020079316-appb-100005
    Figure PCTCN2020079316-appb-100005
    其中，
    Figure PCTCN2020079316-appb-100006
    是双目相机坐标系相对于虚拟空间的世界坐标系的位姿关系，
    Figure PCTCN2020079316-appb-100007
    是股骨上的第一视觉标记物坐标系相对于双目相机坐标系的位姿，
    Figure PCTCN2020079316-appb-100008
    是CT坐标系相对于第一视觉标记物坐标系的位姿，P CT是所述虚拟股骨在CT坐标系下的位姿。     among them,
    Figure PCTCN2020079316-appb-100006
    Is the pose relationship between the binocular camera coordinate system and the world coordinate system of the virtual space,
    Figure PCTCN2020079316-appb-100007
    Is the pose of the coordinate system of the first visual marker on the femur relative to the coordinate system of the binocular camera,
    Figure PCTCN2020079316-appb-100008
    Is the pose of the CT coordinate system relative to the coordinate system of the first visual marker, and P CT is the pose of the virtual femur in the CT coordinate system.
PCT/CN2020/079316 2019-06-18 2020-03-13 Augmented reality navigation method and system for minimally invasive total knee replacement surgery WO2020253280A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910527900.4A CN110353806B (en) 2019-06-18 2019-06-18 Augmented reality navigation method and system for minimally invasive total knee replacement surgery
CN201910527900.4 2019-06-18

Publications (1)

Publication Number Publication Date
WO2020253280A1 true WO2020253280A1 (en) 2020-12-24

Family

ID=68216318

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/079316 WO2020253280A1 (en) 2019-06-18 2020-03-13 Augmented reality navigation method and system for minimally invasive total knee replacement surgery

Country Status (2)

Country Link
CN (1) CN110353806B (en)
WO (1) WO2020253280A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11439469B2 (en) 2018-06-19 2022-09-13 Howmedica Osteonics Corp. Virtual guidance for orthopedic surgical procedures

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110353806B (en) * 2019-06-18 2021-03-12 北京航空航天大学 Augmented reality navigation method and system for minimally invasive total knee replacement surgery
CN110742700B (en) * 2019-11-13 2021-07-30 北京国润健康医学投资有限公司 Simulated weight-bearing brace positioning device and method for augmented reality surgery system
CN111261265B (en) * 2020-01-14 2024-02-27 上海联影医疗科技股份有限公司 Medical imaging system based on virtual intelligent medical platform
CN111292306A (en) * 2020-02-04 2020-06-16 北京航空航天大学 Knee joint CT and MR image fusion method and device
CN111281540B (en) * 2020-03-09 2021-06-04 北京航空航天大学 Real-time visual navigation system based on virtual-actual fusion in minimally invasive surgery of orthopedics department
CN111445531B (en) * 2020-03-24 2022-08-30 云南电网有限责任公司楚雄供电局 Multi-view camera navigation method, device, equipment and storage medium
CN111833405B (en) * 2020-07-27 2023-12-08 北京大华旺达科技有限公司 Calibration and identification method and device based on machine vision
CN112190328A (en) * 2020-09-17 2021-01-08 常州锦瑟医疗信息科技有限公司 Holographic perspective positioning system and positioning method
CN112545649B (en) * 2020-12-02 2022-03-25 中国科学院自动化研究所 Femoral head core decompression operation navigation implementation system based on mixed reality
CN112826590A (en) * 2021-02-02 2021-05-25 复旦大学 Knee joint replacement spatial registration system based on multi-modal fusion and point cloud registration
CN113129451B (en) * 2021-03-15 2022-09-30 北京航空航天大学 Holographic three-dimensional image space quantitative projection method based on binocular vision positioning
CN113081272B (en) * 2021-03-22 2023-02-03 珞石（北京）科技有限公司 Knee joint replacement surgery auxiliary positioning system guided by virtual wall
CN113180828B (en) * 2021-03-25 2023-05-12 北京航空航天大学 Surgical robot constraint motion control method based on rotation theory
CN113129372B (en) * 2021-03-29 2023-11-03 深圳清元文化科技有限公司 Hololens space mapping-based three-dimensional scene semantic analysis method
CN113012230B (en) * 2021-03-30 2022-09-23 华南理工大学 Method for placing surgical guide plate under auxiliary guidance of AR in operation
CN113116523B (en) * 2021-04-09 2022-02-11 骨圣元化机器人(深圳)有限公司 Orthopedic surgery registration device, terminal equipment and storage medium
CN113616350B (en) * 2021-07-16 2022-04-19 元化智能科技(深圳)有限公司 Verification method and device for selected positions of marking points, terminal equipment and storage medium
CN113679473A (en) * 2021-08-23 2021-11-23 北京航空航天大学 Human-computer cooperative force feedback ventricular puncture robot device
CN113855236B (en) * 2021-09-03 2022-05-31 北京长木谷医疗科技有限公司 Method and system for tracking and moving surgical robot
CN114224508A (en) * 2021-11-12 2022-03-25 苏州微创畅行机器人有限公司 Medical image processing method, system, computer device and storage medium
CN114587650B (en) * 2022-02-06 2024-06-11 上海诠视传感技术有限公司 Tooth root canal orifice treatment auxiliary navigation method and system based on mixed reality technology
CN114587657B (en) * 2022-02-06 2024-05-31 上海诠视传感技术有限公司 Auxiliary navigation method and system for oral implantation based on mixed reality technology
CN116993794B (en) * 2023-08-02 2024-05-24 德智鸿(上海)机器人有限责任公司 Virtual-real registration method and device for augmented reality surgery assisted navigation
CN118121312A (en) * 2024-05-06 2024-06-04 北京壹点灵动科技有限公司 Surgical robot system, computer-readable storage medium, and electronic device

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105931237A (en) * 2016-04-19 2016-09-07 北京理工大学 Image calibration method and system
CN106780459A (en) * 2016-12-12 2017-05-31 华中科技大学 A kind of three dimensional point cloud autoegistration method
CN107536643A (en) * 2017-08-18 2018-01-05 北京航空航天大学 A kind of augmented reality operation guiding system of Healing in Anterior Cruciate Ligament Reconstruction
US20180185100A1 (en) * 2017-01-03 2018-07-05 Mako Surgical Corp. Systems And Methods For Surgical Navigation
CN108888341A (en) * 2018-04-26 2018-11-27 上海交通大学 A kind of scaling method of augmented reality Helmet Mounted Display position real-time tracking
CN109659024A (en) * 2018-12-12 2019-04-19 黑龙江拓盟科技有限公司 A kind of remote diagnosis method of MR auxiliary
CN109674532A (en) * 2019-01-25 2019-04-26 上海交通大学医学院附属第九人民医院 Operation guiding system and its equipment, method and storage medium based on MR
CN110353806A (en) * 2019-06-18 2019-10-22 北京航空航天大学 Augmented reality navigation methods and systems for the operation of minimally invasive total knee replacement

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104434313B (en) * 2013-09-23 2019-03-01 中国科学院深圳先进技术研究院 A kind of abdominal surgery navigation methods and systems
US10154239B2 (en) * 2014-12-30 2018-12-11 Onpoint Medical, Inc. Image-guided surgery with surface reconstruction and augmented reality visualization
CN109674534A (en) * 2017-10-18 2019-04-26 深圳市掌网科技股份有限公司 A kind of surgical navigational image display method and system based on augmented reality

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105931237A (en) * 2016-04-19 2016-09-07 北京理工大学 Image calibration method and system
CN106780459A (en) * 2016-12-12 2017-05-31 华中科技大学 A kind of three dimensional point cloud autoegistration method
US20180185100A1 (en) * 2017-01-03 2018-07-05 Mako Surgical Corp. Systems And Methods For Surgical Navigation
CN107536643A (en) * 2017-08-18 2018-01-05 北京航空航天大学 A kind of augmented reality operation guiding system of Healing in Anterior Cruciate Ligament Reconstruction
CN108888341A (en) * 2018-04-26 2018-11-27 上海交通大学 A kind of scaling method of augmented reality Helmet Mounted Display position real-time tracking
CN109659024A (en) * 2018-12-12 2019-04-19 黑龙江拓盟科技有限公司 A kind of remote diagnosis method of MR auxiliary
CN109674532A (en) * 2019-01-25 2019-04-26 上海交通大学医学院附属第九人民医院 Operation guiding system and its equipment, method and storage medium based on MR
CN110353806A (en) * 2019-06-18 2019-10-22 北京航空航天大学 Augmented reality navigation methods and systems for the operation of minimally invasive total knee replacement

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11439469B2 (en) 2018-06-19 2022-09-13 Howmedica Osteonics Corp. Virtual guidance for orthopedic surgical procedures
US11478310B2 (en) 2018-06-19 2022-10-25 Howmedica Osteonics Corp. Virtual guidance for ankle surgery procedures
US11571263B2 (en) 2018-06-19 2023-02-07 Howmedica Osteonics Corp. Mixed-reality surgical system with physical markers for registration of virtual models
US11645531B2 (en) 2018-06-19 2023-05-09 Howmedica Osteonics Corp. Mixed-reality surgical system with physical markers for registration of virtual models
US11657287B2 (en) 2018-06-19 2023-05-23 Howmedica Osteonics Corp. Virtual guidance for ankle surgery procedures
US12020801B2 (en) 2018-06-19 2024-06-25 Howmedica Osteonics Corp. Virtual guidance for orthopedic surgical procedures

Also Published As

Publication number Publication date
CN110353806A (en) 2019-10-22
CN110353806B (en) 2021-03-12

Similar Documents

Publication Publication Date Title
WO2020253280A1 (en) Augmented reality navigation method and system for minimally invasive total knee replacement surgery
JP2022133440A (en) Systems and methods for augmented reality display in navigated surgeries
JP6685580B2 (en) System and method for intraoperative image analysis
US11710246B2 (en) Skin 3D model for medical procedure
JP6463038B2 (en) Image alignment apparatus, method and program
US11490986B2 (en) System and method for improved electronic assisted medical procedures
JP7202737B2 (en) Tracking method and apparatus for dental implant navigation surgery
Hu et al. Head-mounted augmented reality platform for markerless orthopaedic navigation
Zeng et al. A surgical robot with augmented reality visualization for stereoelectroencephalography electrode implantation
CN110638525B (en) Operation navigation system integrating augmented reality
JP6493885B2 (en) Image alignment apparatus, method of operating image alignment apparatus, and image alignment program
Chen et al. Tissue structure updating for in situ augmented reality navigation using calibrated ultrasound and two-level surface warping
US10383692B1 (en) Surgical instrument guidance system
TWI679960B (en) Surgical instrument guidance system
US20240081921A1 (en) System and method for verification of conversion of locations between coordinate systems
CN111658142A (en) MR-based focus holographic navigation method and system
CN114283179A (en) Real-time fracture far-near end space pose acquisition and registration system based on ultrasonic images
Xie et al. A small-scaled intraoperative 3d visualization navigation system for femoral head repair surgery
JP6392190B2 (en) Image registration device, method of operating image registration device, and program
US20230116175A1 (en) System and method for a target reconstruction process
KR102534981B1 (en) System for alignmenting patient position and monitoring with surface image guidance
US20240180629A1 (en) System and method for improved electronic assisted medical procedures
TW201211937A (en) Human face matching system and method thereof
Gooroochurn et al. Preoperative to intraoperative space registration for management of head injuries
WO2023039596A1 (en) Integrated surgical navigation and visualization system, and methods thereof

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20827645

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20827645

Country of ref document: EP

Kind code of ref document: A1