CN116492052B

CN116492052B - Three-dimensional visual operation navigation system based on mixed reality backbone

Info

Publication number: CN116492052B
Application number: CN202310450119.8A
Authority: CN
Inventors: 吕海; 张伟聪; 黄成颉; 苏志海; 陈子扬; 王云飞; 范荣爱
Original assignee: Zhongke Zhibo Zhuhai Technology Co ltd
Current assignee: Zhongke Zhibo Zhuhai Technology Co ltd
Priority date: 2023-04-24
Filing date: 2023-04-24
Publication date: 2024-04-23
Anticipated expiration: 2043-04-24
Also published as: CN116492052A

Abstract

The embodiment of the invention provides a three-dimensional visual surgical navigation system based on a mixed reality spine, which comprises the following components: the system comprises mixed reality equipment provided with a first navigation unit, C-arm imaging equipment provided with a second navigation unit, an optical surgery positioning navigator and a cross-mode image generation module; the optical surgery positioning navigator is used for establishing a three-dimensional space coordinate system according to the coordinate information of the first navigation unit and the coordinate information of the second navigation unit and dynamically tracking the three-dimensional space coordinate of the mixed reality equipment; the cross-modal image generation module is used for generating three-dimensional spine focus fusion image data and sending the three-dimensional spine focus fusion image data to the mixed reality equipment, and the mixed reality equipment is used for carrying out registration processing according to the real-time coordinate information, the position information, the real scene image and the three-dimensional spine focus fusion image to obtain a mixed reality three-dimensional image. The system can reduce human registration errors, improve navigation accuracy, thereby improving shortening operation time and reducing doctor-patient radiation hazard.

Description

Three-dimensional visual operation navigation system based on mixed reality backbone

Technical Field

The invention relates to the technical field of data processing, in particular to a three-dimensional visual surgical navigation system based on a mixed reality spine.

Background

Prolapse of lumbar intervertebral disc is the most common orthopedic disease with a incidence of about 20%, of which 10-15% of patients require surgical treatment. With the aging population and the change of modern people's working and living habits, the number of patients requiring surgery is increasing at a rate of 20% per year. The current surgical treatment method mostly needs to cut the skin at the pathological change position, separate muscle and fascia, then cut off partial structures (such as vertebral lamina, articular process, joint capsule, ligament, etc.) forming the vertebral canal to different degrees, open a window on the vertebral canal wall, finally expose and cut off the pathological change intervertebral disc in the vertebral canal. The method can damage the lumbar vertebra structure to different degrees, causes sequelae such as lumbar vertebra degeneration, lumbar vertebra instability and the like, and even reconstructs the stability of the spine after operation by a fusion internal fixation method. The spine operation navigation system is gradually applied to clinic in recent years due to the advantages of precision, minimally invasive and the like, but has certain limitations: the traditional screen type operation navigation system generally presents different two-dimensional interfaces of lesion positions such as a transverse position, a sagittal position and a coronal position to a surgeon, the relative positions of each tissue structure and the operation instrument lack visual three-dimensional space display and feeling for the operator, the operator is required to reconstruct three-dimensional images in the brain by himself, extremely high requirements are provided for the space imagination capability of the operator, and the technical learning curve is extremely steep for young operators; in addition, as the interface of the navigation system is separated from the operation area, the sight of the operator needs to be frequently switched back and forth between the navigation screen and the operation area, the operation efficiency is reduced, the accuracy of the navigation operation is easily reduced due to improper hand-eye coordination, and the operation effect is influenced.

Disclosure of Invention

The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

The invention mainly aims to provide a three-dimensional visual operation navigation system based on a mixed reality spine, which can reduce human registration errors and improve navigation precision, so that the treatment effect of lumbar disc herniation is improved, the occurrence of complications is reduced, the operation time is shortened, and the doctor-patient radiation hazard is reduced.

In a first aspect, an embodiment of the present invention provides a three-dimensional visualization surgical navigation system based on a mixed reality spine, including:

A mixed reality device comprising a first navigation unit;

A C-arm imaging device comprising a second navigation unit for acquiring radiographic image data of a patient's spine during a surgical procedure;

The optical surgery positioning navigator is used for establishing a three-dimensional space coordinate system according to the acquired coordinate information of the first navigation unit and the coordinate information of the second navigation unit in the process of scanning the spine of the patient, and dynamically tracking the three-dimensional space coordinate of the mixed reality equipment to obtain real-time coordinate information of the mixed reality equipment in the process of surgery;

The cross-modal image generation module is used for generating three-dimensional spine focus fusion image data and sending the three-dimensional spine focus fusion image data to the mixed reality equipment, wherein the three-dimensional spine focus fusion image data comprises a three-dimensional spine focus fusion image and position information in the three-dimensional spine focus fusion image, the three-dimensional spine focus fusion image is an image obtained by registering and fusing the ray image data and first image data, the first image data is an image obtained by carrying out three-dimensional reconstruction on a lumbar vertebra lesion part on a CT image and/or an MRI image of a spine of a patient acquired before an operation, and the position information is obtained according to coordinate information of the C-arm imaging equipment and the three-dimensional spine focus fusion image;

the mixed reality device is further used for acquiring a real scene image, and performing registration processing according to the real-time coordinate information, the position information, the real scene image and the three-dimensional spine focus fusion image to obtain a mixed reality three-dimensional image.

In some alternative embodiments, the first navigation unit includes a first guide flight carrier and a plurality of infrared reflective spheres disposed on the first guide flight carrier, and the second navigation unit includes a second guide flight carrier and a plurality of infrared reflective spheres disposed on the second guide flight carrier.

In some optional embodiments, the mixed reality device is further configured to perform a point cloud registration process on the real scene image and the three-dimensional spine focus fusion image to obtain a three-dimensional image of the mixed reality spine;

the mixed reality device is further used for obtaining a mixed reality three-dimensional image according to the real-time coordinate information, the position information and the mixed reality spine three-dimensional image.

In some optional embodiments, the mixed reality device is further configured to obtain first iteration output data of a point cloud registration algorithm model, source point cloud data and target point cloud data, where the source point cloud data is point cloud data extracted from a patient skin region image of the three-dimensional spine focus fusion image, and the target point cloud data is point cloud data extracted from a patient skin region image of the real scene image;

The mixed reality device is further configured to input the first output data, the source point cloud data, and the target point cloud data to the point cloud registration algorithm model, so as to output second iteration output data through the point cloud registration algorithm model until the second iteration output data meets a preset convergence judgment standard, where the first iteration output data is last output data of the second iteration output data;

the point cloud registration algorithm model comprises a first rigid body transformation unit, a second rigid body transformation unit, a first feature extraction unit, a second feature extraction unit and a calculation matching matrix unit, wherein the first rigid body transformation unit is used for carrying out first rigid body transformation processing on the first iterative output data and the source point cloud data to obtain source point cloud data after rigid body transformation, the second rigid body transformation unit is used for carrying out second rigid body transformation processing on the source point cloud data after rigid body transformation and the target point cloud data to obtain target point cloud data after rigid body transformation, the first feature extraction unit is used for carrying out feature extraction on the source point cloud data after rigid body transformation to obtain source point cloud feature data, the second feature extraction unit is used for carrying out feature extraction on the target point cloud data to obtain target point cloud feature data, the calculation matching matrix unit is used for carrying out calculation processing on the target point cloud data after rigid body transformation, the source point cloud feature data and the target point cloud feature data to obtain matching matrix, and the calculation matching matrix unit is also used for carrying out second iterative output data after rigid body transformation.

In some optional embodiments, the cross-modal image generating module further includes a surgical path calculating unit, where the surgical path calculating unit is configured to perform identification processing on the first image data to obtain a nerve compression target, and calculate a puncture path in the first image data according to the nerve compression target.

In some optional embodiments, the cross-mode image display module further includes an image fusion module unit and a data transmission unit, where the image fusion module unit is configured to perform vertebra segmentation identification processing on the CT image and/or the MRI image of the spine of the patient acquired before the operation to obtain the first image data;

The image fusion module unit is also used for carrying out symmetrical alignment and centroid alignment treatment on the radiographic image data of the patient spine and the first image data acquired in the operation process to obtain aligned primary fusion image data;

The image fusion module unit is also used for carrying out fusion processing on the primary fusion image data by utilizing a point cloud precision registration ICP algorithm to obtain a three-dimensional spine focus fusion image;

the data transmission unit is used for transmitting the three-dimensional spine focus fusion image data generated by the image fusion module unit to the mixed reality equipment.

In some alternative embodiments, the first image comprises a CT image and an MRI image;

the image fusion module unit is further used for performing first positioning processing on the CT image and the MRI image through a vertebra positioning network respectively to obtain positioning point information corresponding to each vertebra of the CT image and positioning point information corresponding to each vertebra in the MRI image;

Dividing each vertebra in the CT image and the MRI image through the positioning point information respectively to obtain first vertebra image data of each vertebra in the CT image and second vertebra image data of each vertebra in the MRI image;

obtaining the CT image and the MRI image after vertebrae segmentation according to the first vertebrae image data of each vertebrae in the CT image and the second vertebrae image data of each vertebrae in the MRI image;

and registering the CT image and the MRI image after vertebrae segmentation to obtain the first image data.

In some alternative embodiments, the image fusion module unit is further configured to:

Extracting l andmark points of the CT image and l andmark points of the MRI image after vertebrae segmentation;

sequentially taking the l andmark points of the CT image and the l andmark points of the MRI image with the same number;

Sequentially aligning the l andmark points of the CT image with the l andmark points of the MRI image in a mass center manner to obtain an initial alignment image;

And performing rigid transformation and pyramid-based non-rigid registration processing on the initial alignment image to obtain the first image data.

In a second aspect, an embodiment of the present invention provides a three-dimensional visualization display method for a mixed reality spine, which is characterized in that the method is applied to the three-dimensional visualization surgical navigation system for a mixed reality spine according to the first aspect, and the method includes:

Acquiring CT images and/or MRI images of the spine of a patient before an operation;

Performing three-dimensional reconstruction on the CT image and/or the MRI image to obtain first image data;

acquiring radiographic image data by the C-arm imaging device;

Registering and fusing the ray image data and the first image data to obtain a three-dimensional spine focus fused image;

acquiring coordinate information of the mixed reality equipment and coordinate information of the C-arm imaging equipment through the first navigation unit and the second navigation unit respectively;

The method comprises the steps that an optical operation positioning navigator is used for establishing a three-dimensional space coordinate system according to the acquired coordinate information of a first navigation unit and the coordinate information of a second navigation unit in the process of scanning the spine of a patient, and dynamically tracking the three-dimensional space coordinate of the mixed reality equipment to obtain real-time coordinate information of the mixed reality equipment in the process of operation;

Processing the three-dimensional spine focus fusion image according to the coordinate information of the C-arm imaging equipment to obtain the position information in the three-dimensional spine focus fusion image;

Acquiring a real scene image through the mixed reality equipment;

And carrying out registration processing according to the real-time coordinate information, the position information, the real scene image and the three-dimensional spine focus fusion image to obtain a mixed reality three-dimensional image.

In a third aspect, an embodiment of the present invention provides a controller, including: the system comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and is characterized in that the processor realizes the three-dimensional visualization display method of the mixed reality spine according to the second aspect when executing the computer program.

In a fourth aspect, a computer-readable storage medium stores computer-executable instructions for performing the method for three-dimensional visualization of a mixed reality spine according to the second aspect.

The beneficial effects of the invention include: a three-dimensional visualization surgical navigation system based on a mixed reality spine, the system comprising: the system comprises mixed reality equipment, C-arm imaging equipment, an optical surgery positioning navigator and a cross-mode image generation module; wherein the mixed reality device comprises a first navigation unit; the C-arm imaging device comprises a second navigation unit and is used for acquiring radiographic image data of the spine of the patient in the operation process; the optical operation positioning navigator is used for establishing a three-dimensional space coordinate system according to the acquired coordinate information of the first navigation unit and the coordinate information of the second navigation unit in the process of scanning the spine of the patient, and dynamically tracking the three-dimensional space coordinates of the mixed reality equipment to obtain real-time coordinate information of the mixed reality equipment in the operation process; the cross-mode image generation module is used for generating three-dimensional spine focus fusion image data and sending the three-dimensional spine focus fusion image data to the mixed reality equipment, wherein the three-dimensional spine focus fusion image data comprises a three-dimensional spine focus fusion image and position information in the three-dimensional spine focus fusion image, the three-dimensional spine focus fusion image is an image obtained by registering and fusing radial image data and first image data, the first image data is an image obtained by carrying out three-dimensional reconstruction on a lumbar vertebra lesion part on a CT image and/or an MRI image of a spine of a patient acquired before an operation, and the position information is obtained according to coordinate information of the C-arm imaging equipment and the three-dimensional spine focus fusion image; the mixed reality device is also used for acquiring a real scene image, and performing registration processing according to the real-time coordinate information, the position information, the real scene image and the three-dimensional spine focus fusion image to obtain a mixed reality three-dimensional image. In the technical scheme of the implementation, based on the three-dimensional visual operation navigation system of the mixed reality spine, the first image data (CT image and/or MRI image of the spine of a patient) before operation and the radiographic image data acquired through the C-arm imaging device during operation can be fused to obtain a three-dimensional spine focus fusion image, and then the three-dimensional spine focus fusion image is registered to the corresponding position of the real scene image through the mixed reality device, so that the precise, rapid and automatic registration of the three-dimensional visual model of the virtual lumbar spine and the real anatomical position of the patient can be realized in the mixed reality device, important structures (such as bones, nerves, blood vessels, ligaments, pathological change intervertebral discs and the like) in the spine of the patient can be dynamically displayed in the real anatomical position of the patient, and doctors can directly and accurately display the important structures in the spine of the patient in the real anatomical position of the patient in the operation process of the mixed reality device, thereby reducing human error, improving navigation precision, improving the treatment effect of lumbar disc herniation, reducing complications, shortening operation time and reducing doctor-patient radiation harm.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

FIG. 1 is a schematic illustration of a three-dimensional visualization surgical navigation system based on a mixed reality spine provided in one embodiment of the present invention;

FIG. 2 is a schematic diagram of a controller in a cross-modality image display module of a mixed reality spinal three-dimensional visualization surgical navigation system provided in accordance with one embodiment of the present invention;

FIG. 3 is a flow chart of a method for three-dimensional visualization of a mixed reality spine for use with a three-dimensional visualization surgical navigation system based on the mixed reality spine provided in one embodiment of the present invention;

FIG. 4 is a schematic view of a CT image provided in accordance with one embodiment of the present invention;

FIG. 5 is a schematic illustration of an MRI image provided in accordance with an embodiment of the present invention;

FIG. 6 is a schematic illustration of a vertebrae positioning network for vertebrae segmentation processing of CT images, MRI images, according to an embodiment of the present invention;

FIG. 7 is a schematic illustration of radiographic image data provided by one embodiment of the invention;

FIG. 8 is a schematic representation of a three-dimensional spinal lesion fusion image provided in accordance with one embodiment of the present invention;

FIG. 9 is a schematic diagram of the operation principle of a mixed reality spinal three-dimensional visualization surgical navigation system provided by an embodiment of the present invention;

Fig. 10 is a schematic diagram of a point cloud registration algorithm model in a three-dimensional visualization display method of a mixed reality spine according to an embodiment of the invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

It should be noted that although functional block division is performed in a device diagram and a logic sequence is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the block division in the device, or in the flowchart. The terms first, second and the like in the description, in the claims and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

In order to solve the above-mentioned problems, an embodiment of the present invention provides a three-dimensional visualization surgical navigation system based on a mixed reality spine, which includes: the system comprises mixed reality equipment, C-arm imaging equipment, an optical surgery positioning navigator and a cross-mode image generation module; wherein the mixed reality device comprises a first navigation unit; the C-arm imaging device comprises a second navigation unit and is used for acquiring radiographic image data of the spine of the patient in the operation process; the optical operation positioning navigator is used for establishing a three-dimensional space coordinate system according to the acquired coordinate information of the first navigation unit and the coordinate information of the second navigation unit in the process of scanning the spine of the patient, and dynamically tracking the three-dimensional space coordinates of the mixed reality equipment to obtain real-time coordinate information of the mixed reality equipment in the operation process; the cross-mode image generation module is used for generating three-dimensional spine focus fusion image data and sending the three-dimensional spine focus fusion image data to the mixed reality equipment, wherein the three-dimensional spine focus fusion image data comprises a three-dimensional spine focus fusion image and position information in the three-dimensional spine focus fusion image, the three-dimensional spine focus fusion image is an image obtained by registering and fusing radial image data and first image data, the first image data is an image obtained by carrying out three-dimensional reconstruction on a lumbar vertebra lesion part on a CT image and/or an MRI image of a spine of a patient acquired before an operation, and the position information is obtained according to coordinate information of the C-arm imaging equipment and the three-dimensional spine focus fusion image; the mixed reality device is also used for acquiring a real scene image, and performing registration processing according to the real-time coordinate information, the position information, the real scene image and the three-dimensional spine focus fusion image to obtain a mixed reality three-dimensional image.

In the technical scheme of the embodiment, in operation, a patient spine is scanned through a C-arm imaging device to obtain ray image data of the patient spine, a cross-mode image display module registers and fuses the ray image data and the first image data to generate a three-dimensional spine focus fusion image, a three-dimensional space coordinate system is built through an optical operation positioning navigator according to the obtained coordinate information of a first navigation unit and the coordinate information of a second navigation unit in the process of scanning the patient spine, dynamic tracking is carried out on three-dimensional space coordinates of a mixed reality device to obtain real-time coordinate information of the mixed reality device in the operation process, and because the three-dimensional spine focus fusion image is obtained according to the ray image data obtained by the C-arm imaging device, the coordinate information of the C-arm imaging device when obtaining the image can be matched with the coordinate on the three-dimensional spine focus fusion image, and the three-dimensional space coordinate system is built according to the obtained coordinate information of the first navigation unit in the process of scanning the patient spine and the coordinate information of the second navigation unit on the C-arm imaging device, so that the three-dimensional scene focus fusion image can be matched with the real-dimensional scene fusion image according to the real-time coordinate information of the mixed reality device and the three-dimensional space coordinate system. In the technical scheme of the implementation, based on the three-dimensional visual operation navigation system of the mixed reality spine, the first image data (CT image and/or MRI image of the spine of a patient) before operation and the radiographic image data acquired through the C-arm imaging device during operation can be fused to obtain a three-dimensional spine focus fusion image, and then the three-dimensional spine focus fusion image is registered to the corresponding position of the real scene image through the mixed reality device, so that the precise, rapid and automatic registration of the three-dimensional visual model of the virtual lumbar spine and the real anatomical position of the patient can be realized in the mixed reality device, important structures (such as bones, nerves, blood vessels, ligaments, pathological change intervertebral discs and the like) in the spine of the patient can be dynamically displayed in the real anatomical position of the patient, and doctors can directly and accurately display the important structures in the spine of the patient in the real anatomical position of the patient in the operation process of the mixed reality device, thereby reducing human error, improving navigation precision, improving the treatment effect of lumbar disc herniation, reducing complications, shortening operation time and reducing doctor-patient radiation harm.

Embodiments of the present invention will be further described below with reference to the accompanying drawings.

Referring to fig. 1, fig. 1 is a three-dimensional visualization surgical navigation system based on a mixed reality spine according to an embodiment of the present invention, including: the mixed reality device 110, the C-arm imaging device 120, the optical surgery positioning navigator 130 and the cross-modal image generation module 140, wherein the mixed reality device 110 comprises a first navigation unit 111, the C-arm imaging device 120 comprises a second navigation unit, the C-arm imaging device 120 is used for acquiring radiographic image data of a patient spine in a surgery process, the optical surgery positioning navigator 130 is used for acquiring coordinate information of the first navigation unit 111 and coordinate information of the second navigation unit, and a three-dimensional space coordinate system is established according to the acquired coordinate information of the first navigation unit 111 and the acquired coordinate information of the second navigation unit, so that three-dimensional space coordinates of the mixed reality device 110 are dynamically tracked through the three-dimensional space coordinate system, and real-time coordinate information of the mixed reality device 110 is obtained. The cross-mode image generation module 140 is configured to generate three-dimensional spine focus fusion image data and send the three-dimensional spine focus fusion image data to the mixed reality device 110, where the three-dimensional spine focus fusion image data includes a three-dimensional spine focus fusion image and position information in the three-dimensional spine focus fusion image, the three-dimensional spine focus fusion image is an image obtained by registering and fusing radial image data and first image data, the first image data is an image obtained by three-dimensionally reconstructing a CT image and/or an MRI image of a spine of a patient obtained before an operation, and the position information is obtained according to coordinate information of the C-arm imaging device 120 and the three-dimensional spine focus fusion image.

In some alternative embodiments, the first navigation unit 111 includes a first guide space frame and a plurality of infrared reflection balls disposed on the first guide space frame, the second navigation unit includes a second guide space frame and a plurality of infrared reflection balls disposed on the second guide space frame, the optical surgical positioning navigator 130 is a NDIVEGA optical positioner, and the position of the mixed reality device 110 can be dynamically tracked by the NDI VEGA optical positioner to obtain real-time coordinate information of the mixed reality device 110, so that a relative relationship between a real scene image acquired by the mixed reality device 110 and a three-dimensional spine focus fusion image fused with radiation image data acquired by the C-arm imaging device 120 can be determined according to the real-time coordinate information of the mixed reality device 110 and the position information of the three-dimensional spine focus fusion image. It is understood that the plurality of infrared-reflection balls represents at least three infrared-reflection balls, which are not particularly limited in this embodiment.

In some alternative embodiments, the infrared reflection device (the first navigation unit 111) is connected and assembled with Microsoft HoloLens head-mounted MR equipment (the mixed reality equipment 110), holonens 2 glasses for short, an NDI VEGA optical locator (the optical surgery positioning navigator 130) is adopted to dynamically track three-dimensional space coordinates of the holonens 2 glasses (the mixed reality equipment 110), an inertial tracking system, an accelerometer and a gyroscope in holonens 2 are called to ensure that the content of holographic image presentation in the MR field of view can be changed correspondingly along with the movement of the head of a surgeon to the conversion of the MR viewing angle, the spatial position of the virtual lumbar three-dimensional model does not move severely, and always keeps high coincidence with the real anatomical position under optical tracking, so that the occurrence of the reduction of navigation surgery accuracy caused by image drift generated by the large-amplitude movement of the head of the surgeon in surgery is prevented. Further, the optical tracking device, namely the related infrared transmitting and receiving device, can be completely integrated on the head-mounted MR device, so that the traditional binocular camera can be separated, the operating room space occupied by the navigation device is reduced, and the integrated operation of the mixed reality and the navigation positioning with more intelligent integration is realized.

It should be noted that, under the condition of selecting Microsoft HoloLens as the mixed reality MR technology deployment device, a computer workstation based on the ARM64 architecture is established through the UWP platform, and corresponding development software is deployed on the workstation; installing Win10 SDK 10.0.18362.0, performing format conversion on the lumbar vertebra multi-mode fusion image and planned operation path data, importing the data into a Unity 2019.4 environment for MR scene construction, compiling and generating a C# project by using a MRTK (Mixed Reality ToolKit) 2.4.0 tool kit, deploying the generated C# solution on Microsoft HoloLens2 through Microsoft Visual Studio 2019, wearing Microsoft HoloLens2 by an operator, starting a deployed application program, and then viewing a virtual lumbar vertebra three-dimensional visualization model for virtual-reality space entity interaction.

In some optional embodiments, the mixed reality device 110 is further configured to perform a point cloud registration process on the real scene image and the three-dimensional spine focus fusion image to obtain a three-dimensional image of the mixed reality spine; and then obtaining the mixed reality three-dimensional image according to the real-time coordinate information, the position information and the mixed reality spine three-dimensional image. In an actual use scene, as the mixed reality device 110 is worn by medical staff and moves in the process of an operation, the mixed reality three-dimensional image is obtained through real-time coordinate information, position information and the mixed reality spine three-dimensional image, the position of the mixed reality spine three-dimensional image in the image can not change along with the movement of the mixed reality device 110, and the three-dimensional spine focus fusion image can be fixed on a target spine region of a patient in a real scene image, so that the registration accuracy can be improved, and the problem that deviation occurs in point cloud registration between the real scene image and the three-dimensional spine focus fusion image due to the movement of the head of the medical staff wearing the mixed reality device 110 in the operation process is solved.

Specifically, the method for point cloud registration processing comprises the following steps: acquiring first iteration output data, source point cloud data and target point cloud data of a point cloud registration algorithm model, wherein the source point cloud data is point cloud data extracted from a patient skin area image of a three-dimensional spine focus fusion image, and the target point cloud data is point cloud data extracted from a patient skin area image of a real scene image; the mixed reality device is further configured to input the first output data, the source point cloud data and the target point cloud data to a point cloud registration algorithm model, so as to output second iteration output data through the point cloud registration algorithm model until the second iteration output data meets a preset convergence judgment standard, where the first iteration output data is last output data of the second iteration output data; the point cloud registration algorithm model comprises a first rigid body transformation unit, a second rigid body transformation unit, a first feature extraction unit, a second feature extraction unit and a calculation matching matrix unit, wherein the first rigid body transformation unit is used for carrying out first rigid body transformation processing on first iteration output data and source point cloud data to obtain source point cloud data after rigid body transformation, the second rigid body transformation unit is used for carrying out second rigid body transformation processing on the source point cloud data after rigid body transformation and target point cloud data to obtain target point cloud data after rigid body transformation, the first feature extraction unit is used for carrying out feature extraction on the source point cloud data after rigid body transformation to obtain source point cloud feature data, the second feature extraction unit is used for carrying out feature extraction on the target point cloud data to obtain target point cloud feature data, the calculation matching matrix unit is used for carrying out calculation processing on the target point cloud data after rigid body transformation, the source point cloud feature data and the target point cloud feature data to obtain a matching matrix, and the calculation matching matrix unit is also used for carrying out registration processing on the target point cloud data through the matching matrix to obtain second iteration output data.

In some optional embodiments, the cross-modality image generation module 140 is configured to perform vertebra segmentation recognition processing on a CT image and/or an MRI image of a spine of a patient acquired before an operation to obtain first image data; then, the radiographic image data and the first image data of the patient spine acquired in the operation process are subjected to symmetrical alignment and centroid alignment treatment, and initial fusion image data after alignment is obtained; and then, fusing the primary fused image data by utilizing a point cloud fine registration ICP algorithm to obtain a three-dimensional spine focus fused image, and transmitting the three-dimensional spine focus fused image to mixed reality equipment.

In some alternative embodiments, the first image comprises a CT image and an MRI image; because the shape and gray level difference between each vertebra are smaller, the vertebrae are difficult to distinguish when being automatically segmented, although bones can be extracted through threshold processing, the space between each vertebra is communicated and difficult to separate, the segmentation of the vertebrae is classified into multiple types of segmentation of the vertebrae, the aim is to independently segment each vertebra (comprising lumbar bones and sacrum) and joint capsules between the vertebrae, support subsequent puncture planning, navigation and CT and MRI fusion, and then the image fusion module unit can be also used for carrying out first positioning processing on CT images and MRI images by using vertebrae positioning networks respectively to obtain positioning point information corresponding to each vertebra of the CT images and positioning point information corresponding to each vertebra of the MRI images; then, respectively dividing each vertebra in the CT image and the MRI image through positioning point information to obtain first vertebra image data of each vertebra in the CT image and second vertebra image data of each vertebra in the MRI image; obtaining a CT image and an MRI image after vertebrae segmentation according to the first vertebrae image data of each vertebrae in the CT image and the second vertebrae image data of each vertebrae in the MRI image; and registering the CT image and the MRI image after vertebrae segmentation to obtain first image data.

In some alternative embodiments, since the measure of the multi-mode image is difficult to define, the CT/MRI multi-mode image is further automatically registered based on the vertebral segmentation of the CT image and the MRI image, and then the image fusion module unit may be further configured to extract the l andmark points of the CT image and the l andmark points of the MRI image after the vertebral segmentation; then sequentially taking l andmark points of the CT image and l andmark points of the MRI image with the same number; sequentially aligning the l andmark points of the CT image with the l andmark points of the MRI image to obtain an initial aligned image; and then, carrying out rigid transformation and pyramid-based non-rigid registration processing on the initial alignment image to obtain first image data. The registration network adopts a Laplace network, and the network can obtain better performance in a plurality of computer vision tasks such as super-resolution image reconstruction and optical flow field estimation. The network consists of three parts, namely feature coding, a plurality of residual error modules and feature decoding. The method comprises the steps of taking two 3*3 convolution layers as feature extraction layers, taking one convolution with the step length of 2 as downsampling, taking five residual modules, taking one deconvolution and two continuous convolutions as feature decoding layers, adding long connection, and outputting three-channel deformation vector fields with the same size as the input size.

In some optional embodiments, the cross-mode image generating module 140 further includes a surgical path calculating unit, where the surgical path calculating unit is configured to identify the first image data to obtain a nerve compression target, calculate, according to the nerve compression target, a puncture path in the first image data, and send the puncture path to the display screen unit, where the puncture path is displayed by the display screen unit. Through the operation path calculation unit in the embodiment, the intelligent puncture path planning module of the navigation system needs to automatically identify the diseased herniated lumbar disc and structures such as bones, ligaments, pressed nerves or blood vessels nearby by means of an intelligent algorithm, and provides a safe and effective puncture path scheme for an operating doctor to refer to and select on the premise of not damaging important tissues. The artificial intelligent deep learning algorithm is utilized to automatically learn and divide structures such as vertebrae, intervertebral discs, nerves and blood vessels around lumbar intervertebral disc foramen, the lumbar intervertebral disc protrusion is taken as the center, parameters such as the center of different operation path targets, the adjacent important nerve blood vessels, the joint process joint distance, the corresponding Kambin triangle area and the like are calculated through analysis, the related percutaneous puncture angle and depth are recorded, the convolutional neural network is used for continuous iterative training, and finally, the optimal operation path scheme or schemes are provided for an operator to refer to and select.

Fig. 2 is a schematic diagram of a controller for performing a three-dimensional visualization display method of a mixed reality spine according to an embodiment of the present application, where the controller is disposed in a cross-modal image display module of the three-dimensional visualization surgical navigation system of fig. 1.

In the example of fig. 2, the controller 100 is provided with a processor 110 and a memory 120, wherein the processor 110 and the memory 120 may be connected by a bus or otherwise, in fig. 1 by way of example.

Memory 120, as a non-transitory computer-readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer-executable programs. In addition, memory 120 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 120 optionally includes memory remotely located relative to the processor 110, which may be connected to the controller 100 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The controller may be a programmable controller, or may be other controllers, which are not particularly limited in this embodiment.

It will be appreciated by those skilled in the art that the controller shown in fig. 2 is not limiting of the embodiments of the application and may include more or fewer components than shown, or certain components may be combined, or a different arrangement of components.

Based on the controller, various embodiments of the present invention applied to a three-dimensional visualization display method of a mixed reality spine based on a three-dimensional visualization surgical navigation system of the mixed reality spine are presented below.

Referring to fig. 3, fig. 3 is a flowchart of a three-dimensional visualization display method of a mixed reality spine according to an embodiment of the present invention, and the three-dimensional visualization display method of the mixed reality spine according to the embodiment of the present invention may include, but is not limited to, step S100, step S200, step S300, step S400, step S500, step S600, step S700, step S800, and step S900.

Step S100, CT images and/or MRI images of the spine of the patient before the operation are acquired.

Specifically, the CT image shown in fig. 4 can clearly show bone tissue, and the MRI image shown in fig. 5 can finely show soft tissues such as skeletal muscle, diseased intervertebral disc or ligament, pressed nerve or blood vessel, and the like, so that the diseased condition that cannot be detected by ultrasonic detection can be identified.

Step S200, performing three-dimensional reconstruction on lumbar vertebra lesion parts on the CT image and/or the MRI image to obtain first image data.

Specifically, the fusion image obtained by carrying out three-dimensional reconstruction on the CT image and the MRI image can enable a doctor to recognize the lesions of bones, gases and the like which are not easy to be detected by ultrasonic in the operation before the operation, to be recognized again, and the rest areas which are not displayed by ultrasonic scanning in the same plane can be seen in the fusion image of the CT image and the MRI image at the same time. The cross-modality image fusion technique results in first image data with better resolution and a larger field of view than ultrasound.

In some optional embodiments, performing first positioning processing on the CT image and the MRI image through a vertebra positioning network to obtain positioning point information corresponding to each vertebra of the CT image and positioning point information corresponding to each vertebra in the MRI image; then, respectively dividing each vertebra in the CT image and the MRI image through positioning point information to obtain first vertebra image data of each vertebra in the CT image and second vertebra image data of each vertebra in the MRI image; obtaining a CT image and an MRI image after vertebrae segmentation according to the first vertebrae image data of each vertebrae in the CT image and the second vertebrae image data of each vertebrae in the MRI image; and registering the CT image and the MRI image after vertebrae segmentation to obtain first image data.

In some optional embodiments, the step of performing registration processing on the CT image and the MRI image after the vertebrae are segmented to obtain first image data may first perform registration processing on the CT image and the MRI image after the vertebrae are segmented to obtain first image data, where l andmark points of the CT image and l andmark points of the MRI image after the vertebrae are segmented may first be extracted; then sequentially taking l andmark points of the CT image and l andmark points of the MRI image with the same number; sequentially aligning the l andmark points of the CT image with the l andmark points of the MRI image to obtain an initial aligned image; and then, carrying out rigid transformation and pyramid-based non-rigid registration processing on the initial alignment image to obtain first image data.

In some alternative embodiments, referring to fig. 6, the vertebral positioning network is made up of two parts, the first part being an automatic positioning network of the vertebrae and the second part being the vertebrae where the positioning points are located, by means of automatic positioning points. Each anchor point heat map is regressed in the vertebra positioning network through a semantic segmentation network. The heat map is generated by firstly calculating the mass center x _i of the ith vertebra, and generating a gold standard heat map by adopting a Gaussian blur function according to the mass center:

The closer the representation is to the target point x _i, the larger the gray value, where σi is a learnable parameter, so that the heat map is similar to a spot image, the initially generated heat map HLA is a rough heat map, multiple spots may come out or the spots may be blurred, the spot image is input into several consecutive convolution kernels (shown in green), the goal is to eliminate those false spots, and the larger convolution kernel scale is used to increase the receptive field of the feature, further extracting the target point from the global perspective, and obtaining a more accurate heat map HSC. HLA has the advantages of smaller visual field, local feature extraction, more accurate target points can be predicted, the defect of high false positive aiming at specific target points, the HSC has the advantages of large receptive field and low false positive, but inaccurate predicted position, so that the advantages of the two are combined, and a target heat map is obtained by multiplying:

the objective function employs an L2 loss function:

the final anchor point takes the coordinates of the point in the heat map with the largest gray value:

According to the obtained heat map points, extracting local vertebra original gray level images in a clipping mode, taking the heat map and the gray level images as the input of a segmentation network U-Net, and further focusing on the vertebra where the segmentation positioning points are located, wherein a loss function adopts cross entropy and DSC loss function:

step S300, acquiring radiographic image data by the C-arm imaging apparatus.

Specifically, although the first image data obtained by fusing the CT image and the MRI image can identify the pathological condition, the real-time condition of the patient cannot be obtained, and during the operation, the spine of the patient needs to be scanned by the C-arm imaging device, so as to obtain the real-time radiographic image data as shown in fig. 7.

And step S400, registering and fusing the ray image data and the first image data to obtain a three-dimensional spine focus fused image.

Specifically, the three-dimensional spine focus fusion image shown in fig. 8 is obtained by registering and fusing the radiation image data and the first image data, and under the condition that the three-dimensional spine focus fusion image can obtain the real-time image of the spine of the patient in real time, the focus condition of the patient can be obtained from the CT image and the MRI image before operation. The three-dimensional spine focus fusion image obtained by the cross-mode image fusion technology has the following advantages: 1. overlapping the radiographic image data and the MRI/CT image is equivalent to phase change to increase the definition and scanning range of ultrasound in operation; 2. real-time rapid imaging makes registration adjustment of preoperative and intra-operative images simpler, which is very advantageous for guiding interventional exam treatments.

In some optional embodiments, firstly, performing symmetrical alignment and centroid alignment processing on radiographic image data and first image data of a patient spine acquired in a surgical process to obtain aligned primary fusion image data; the image fusion module unit is also used for carrying out fusion processing on the primary fusion image data by utilizing a point cloud fine registration ICP algorithm to obtain a three-dimensional spine focus fusion image and transmitting the three-dimensional spine focus fusion image to the display screen unit.

And S500, acquiring coordinate information of the mixed reality equipment and coordinate information of the C-arm imaging equipment through the first navigation unit and the second navigation unit respectively.

Specifically, after the generation of the three-dimensional spine focus fusion image is completed, in order to ensure that the three-dimensional spine focus fusion image is registered to the real scene image in the case that the mixed reality device moves in the process of the operation, coordinate information of the mixed reality device and coordinate information of the C-arm imaging device need to be acquired through the first navigation unit and the second navigation unit, respectively.

Step S600, a three-dimensional space coordinate system is established according to the acquired coordinate information of the first navigation unit and the coordinate information of the second navigation unit in the process of scanning the spine of the patient through the optical surgery positioning navigator, and the three-dimensional space coordinates of the mixed reality equipment are dynamically tracked, so that real-time coordinate information of the mixed reality equipment in the surgery process is obtained.

Specifically, after the coordinate information of the first navigation unit and the coordinate information of the second navigation unit are acquired, a three-dimensional space coordinate system of the mixed reality device and the C-arm imaging device is established for the acquired coordinate information of the first navigation unit and the acquired coordinate information of the second navigation unit in the process of scanning the spine of the patient through the optical surgery positioning navigator, and then the mixed reality device is dynamically tracked based on the three-dimensional space coordinate system, so that real-time coordinate information of the mixed reality device is obtained, and the relative position relation between the real scene image and the three-dimensional spine focus fusion image can be determined according to the real-time coordinate information of the mixed reality device and the coordinate information of the C-arm imaging device.

And step S700, obtaining the position information in the three-dimensional spine focus fusion image according to the coordinate information of the C-arm imaging device and the three-dimensional spine focus fusion image.

Specifically, the position information in the three-dimensional spine focus fusion image can be obtained by processing the coordinate information of the C-arm imaging device and the three-dimensional spine focus fusion image, the position information represents the corresponding position information of each pixel point in the three-dimensional spine focus fusion image in a three-dimensional space coordinate system, and the position information in the three-dimensional spine focus fusion image can be associated with the position relation of the image acquired by the mixed reality device because the three-dimensional spine focus fusion image is the image fused based on the ray image data acquired by the C-arm imaging device.

Step S800, acquiring a real scene image by a mixed reality device.

Specifically, in order to register the three-dimensional spine focus fusion image to a real scene facing a mixed reality device worn by a medical staff, a real scene image is acquired through the mixed reality device. In some optional embodiments, in step S500, the mixed reality device may acquire the real scene image simultaneously, the C-arm imaging device may acquire the radiographic image data simultaneously, and a three-dimensional space coordinate system is established by using the coordinate information of the first navigation unit and the coordinate information of the second navigation unit acquired at the time point, so that the coordinate information of each pixel point of the real scene image in the three-dimensional space coordinate system may be obtained, and similarly, the coordinate information of each pixel point of the radiographic image data acquired synchronously by the C-arm imaging device in the three-dimensional space coordinate system may also be obtained, so that the coordinate information of the real scene image acquired by the mixed reality device may be associated with the coordinate information of the radiographic image data in the three-dimensional space, that is, the coordinate information of the real scene image acquired by the mixed reality device may be associated with the position information of the three-dimensional spine focus fusion image.

And step S900, carrying out registration processing according to the real-time coordinate information, the position information, the real scene image and the three-dimensional spine focus fusion image to obtain the mixed reality three-dimensional image.

Specifically, point cloud registration processing is carried out on a real scene image and a three-dimensional spine focus fusion image to obtain a mixed reality spine three-dimensional image; and then obtaining the mixed reality three-dimensional image according to the real-time coordinate information, the position information and the mixed reality spine three-dimensional image.

In some alternative embodiments, as shown in fig. 9-10, the process of performing point cloud registration processing on the real scene image and the three-dimensional spine focus fusion image to obtain the three-dimensional image of the mixed reality spine is specifically as follows: acquiring first iteration output data, source point cloud data and target point cloud data of a point cloud registration algorithm model, wherein the source point cloud data is point cloud data extracted from a patient skin area image of a three-dimensional spine focus fusion image, and the target point cloud data is point cloud data extracted from a patient skin area image of a real scene image; the mixed reality device is further configured to input the first output data, the source point cloud data and the target point cloud data to a point cloud registration algorithm model, so as to output second iteration output data through the point cloud registration algorithm model until the second iteration output data meets a preset convergence judgment standard, where the first iteration output data is last output data of the second iteration output data; the point cloud registration algorithm model comprises a first rigid body transformation unit, a second rigid body transformation unit, a first feature extraction unit, a second feature extraction unit and a calculation matching matrix unit, wherein the first rigid body transformation unit is used for carrying out first rigid body transformation processing on first iteration output data and source point cloud data to obtain source point cloud data after rigid body transformation, the second rigid body transformation unit is used for carrying out second rigid body transformation processing on the source point cloud data after rigid body transformation and target point cloud data to obtain target point cloud data after rigid body transformation, the first feature extraction unit is used for carrying out feature extraction on the source point cloud data after rigid body transformation to obtain source point cloud feature data, the second feature extraction unit is used for carrying out feature extraction on the target point cloud data to obtain target point cloud feature data, the calculation matching matrix unit is used for carrying out calculation processing on the target point cloud data after rigid body transformation, the source point cloud feature data and the target point cloud feature data to obtain a matching matrix, and the calculation matching matrix unit is also used for carrying out registration processing on the target point cloud data through the matching matrix to obtain second iteration output data.

In some alternative embodiments, as shown in fig. 9, (1) the CT and MRI images scanned prior to the operation of the patient are segmented and modeled to obtain model data of the spinal surgical area of the patient, including bones, blood vessels, nerves, target lesion areas, etc., and introduced into the Ho loLens internal operating system; (2) Then Ho loLens is adopted in the operation, and three-dimensional model data in the operation of the region to be operated is obtained by utilizing a three-dimensional reconstruction module of the three-dimensional model data. (3) Registering the model data obtained in the steps (1) and (2) according to the model data to obtain a corresponding transformation relationship between the preoperative multi-mode image space and the intraoperative physical space of the patient; (4) After obtaining the spatial transformation relation between preoperative and intraoperative, mapping important model data such as focus, nerve, blood vessel and the like obtained by preoperative segmentation into physical space (namely a real scene image) of an intraoperative patient after registration spatial transformation; (5) Then, a surgeon wears Ho loLens (mixed reality equipment) during operation, and a visual preoperative spine multi-tissue three-dimensional model can be superimposed under the background of a real operation area of a patient by utilizing the mixed reality rendering function of Ho loLens, so that the aim of performing accurate surgery under mixed reality navigation can be fulfilled.

In the solution of the alternative embodiment described above, the most important is the registration between preoperative and intra-operative data. For this purpose, a point cloud registration algorithm is introduced to obtain the transformation relationship between the preoperative and intraoperative coordinate spaces. As shown in fig. 10, the essence of point cloud registration is that the point cloud data obtained in different coordinate systems are transformed in the coordinate systems, and the key of the problem is that after the point cloud data in one coordinate system is transformed to the other coordinate system through coordinate transformation parameters R (rotation matrix) and T (translation vector), the measured distance between the corresponding points is the smallest, and meanwhile, the problem of the operation efficiency of the registration algorithm is considered, and the problem can be solved by adopting the point cloud registration algorithm based on a deep learning frame. The registration problem of a pair of point cloud data is pairwise registration, and the data set of one point cloud is accurately registered with the data set of the other point cloud (target data set) by applying a4×4 rigid body transformation matrix representing translation and rotation, which is specifically implemented as follows: firstly, selecting a standard from two data sets according to the same key points, and extracting the key points; then calculating the feature descriptors of all the selected key points respectively; then, the coordinate positions of the feature descriptors in the two data sets are combined, the corresponding relation of the feature descriptors and the coordinate positions of the feature descriptors are estimated based on the similarity of the features and the positions of the feature descriptors, and corresponding point pairs are estimated preliminarily; then assuming the data is noisy, the erroneous point pairs that have an effect on registration need to be removed; and estimating the rigid body transformation by using the residual correct corresponding relation to finish registration.

It should be noted that, in the step of extracting the key points, two sets of feature vectors (source point cloud and target point cloud) obtained from the point cloud data need to find similar features, determine overlapping portions of the data, and then register. Different methods are used to search for correspondence between features according to the types of features. The common processing strategies include exhaustion method and KD tree nearest neighbor search, and the embodiment is not limited in particular.

In the step of removing the correspondence, because the influence of noise is not always correct for all estimated correspondences, the incorrect correspondences will have negative influence on the estimation of the final rigid transformation matrix, so that the incorrect correspondences are removed by adopting random sampling consistency estimation, and the number of the final correspondences only uses a certain proportion of correspondences, thereby not only improving the estimation accuracy of the transformation matrix, but also improving the speed of the alignment points.

It should be noted that, for the estimation method of the transformation matrix, firstly, some error metrics are estimated on the basis of the corresponding relationship; then estimating a rigid body transformation under camera pose (motion estimation) and minimization error metrics; then optimizing the structure of the points; then using rigid transformation to rotationally translate the point cloud to the same coordinate system where the target is located, and calculating an internal ICP cycle by using all points, a subset of the points or key points; and then iterating until the convergence judgment standard is met. The convergence determination criterion may be set according to practical situations, and is not specifically limited in this embodiment.

In the technical solution in the above embodiment, the following technical effects can be achieved:

(1) The multi-mode image fusion technology such as CT and MRI before spine operation and three-dimensional C-arm scanning during operation is adopted, a three-dimensional visual model containing bones, muscles, nerves, blood vessels, intervertebral discs and pathological tissues of pathological positions is established, nerve compression sites are presented and serve as radio frequency treatment targets, and a foundation is provided for guiding a radio frequency tool bit to accurately treat the pathological tissues compressing nerves during operation.

(2) Through the collection arrangement to clinical lumbar disc herniation patient big data, adopt artificial intelligence analysis to assist the operation sign of determining, automatic identification pathological change target point, calculate best operation route, under the prerequisite that does not harm and destroy important nerve vascular tissue and the original stable structure of lumbar vertebrae, can accurate radio frequency release nerve oppression.

(3) The method comprises the steps of generating a virtual lumbar vertebra holographic three-dimensional visual model in an actual operation area by utilizing a mixed reality technology, displaying relevant structures such as a herniated disk, nearby bones, nerves and blood vessels in a three-dimensional clear manner under the condition of visible operation field, enabling an operator to more clearly know the relative position and distance relation between a treatment target point and surrounding important structures, simultaneously realizing automatic registration of an MR model and the actual anatomical position of a patient, tracking the position of the patient and dynamic position change of surgical instruments in real time by adopting an optical navigation system, and really realizing accurate radio frequency target treatment on pressed nerve pathological tissues.

(4) The whole set of three-dimensional visual operation navigation system for the mixed reality spine is expected to achieve the purposes of intelligence, precision, minimally invasive, safety and high efficiency, achieves the purposes that the skin incision is only 1-2mm, a patient can walk and discharge the hospital in the same day after operation, introduces minimally invasive operation, digital operation navigation technology, artificial intelligence technology, precision surgery technology and mixed reality technology into lumbar disc herniation operation at the same time, and establishes a new technical platform for lumbar vertebra operation.

Furthermore, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for performing the above-described mixed reality spinal three-dimensional visualization method, for example, performing the above-described method steps S100 to S900 in fig. 3.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any information delivery media. The computer-readable storage medium may be nonvolatile or volatile.

While the preferred embodiment of the present application has been described in detail, the present application is not limited to the above embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit and scope of the present application, and these equivalent modifications or substitutions are included in the scope of the present application as defined in the appended claims.

Claims

1. A three-dimensional visual surgical navigation system based on a mixed reality spine, comprising:

A mixed reality device comprising a first navigation unit;

The mixed reality device is also used for acquiring a real scene image, and carrying out registration processing according to the real-time coordinate information, the position information, the real scene image and the three-dimensional spine focus fusion image to obtain a mixed reality three-dimensional image;

The cross-mode image generation module further comprises an image fusion module unit and a data transmission unit, wherein the image fusion module unit is used for performing vertebrae segmentation recognition processing on the CT image and/or the MRI image of the spine of the patient acquired before an operation to obtain the first image data;

the data transmission unit is used for transmitting the three-dimensional spine focus fusion image data generated by the image fusion module unit to the mixed reality equipment;

The first image comprises a CT image and an MRI image, and the image fusion module unit is further used for performing first positioning processing on the CT image and the MRI image through vertebrae positioning networks respectively to obtain positioning point information corresponding to each vertebra of the CT image and positioning point information corresponding to each vertebra of the MRI image;

registering the CT image and the MRI image after vertebrae segmentation to obtain the first image data;

The heat map of each positioning point is regressed in the vertebra positioning network through a semantic segmentation network, and the heat map generating method is obtained through the following formula:

；

xi is the centroid of the ith vertebra, σi is the learnable parameter:

；

HLA is an initial heat map, HSC is used for eliminating false light spots in the HLA, a convolution kernel scale is used for improving a receptive field of the characteristics, a heat map obtained by a target point is extracted from a global angle, and a target heat map is obtained in a multiplication mode;

the objective function employs an L2 loss function:

；

the locating point takes the coordinates of the point with the maximum gray value in the heat map:

；

Extracting a local vertebra original gray image by a clipping mode according to the coordinates of the points of the obtained heat map, taking the heat map and the gray map as the input of a segmentation network U-Net, and focusing on the vertebra where the segmentation positioning points are located, wherein a loss function adopts cross entropy and DSC loss function:

。

2. The three-dimensional visual surgical navigation system based on a mixed reality spinal column of claim 1, wherein the first navigation unit comprises a first guided aircraft carrier and a plurality of infrared reflective spheres disposed on the first guided aircraft carrier, and the second navigation unit comprises a second guided aircraft carrier and a plurality of infrared reflective spheres disposed on the second guided aircraft carrier.

3. The three-dimensional visual surgical navigation system based on a mixed reality spinal column according to claim 1, wherein,

The mixed reality device is further used for carrying out point cloud registration processing on the real scene image and the three-dimensional spine focus fusion image to obtain a mixed reality spine three-dimensional image;

4. The three-dimensional visual surgical navigation system based on the mixed reality spinal column according to claim 3,

The mixed reality device is further configured to obtain first iteration output data of a point cloud registration algorithm model, source point cloud data and target point cloud data, where the source point cloud data is point cloud data extracted from a patient skin area image of the three-dimensional spine focus fusion image, and the target point cloud data is point cloud data extracted from a patient skin area image of the real scene image;

the mixed reality device is further configured to input the first iteration output data, the source point cloud data, and the target point cloud data to the point cloud registration algorithm model, so as to output second iteration output data through the point cloud registration algorithm model until the second iteration output data meets a preset convergence judgment standard, where the first iteration output data is last output data of the second iteration output data;

5. The three-dimensional visualization surgical navigation system based on the mixed reality spinal column according to claim 1, wherein the cross-modal image generation module further comprises a surgical path calculation unit, the surgical path calculation unit is used for performing recognition processing on first image data to obtain a nerve compression target point, and calculating a puncture path in the first image data according to the nerve compression target point.

6. The mixed reality spine-based three-dimensional visualization surgical navigation system of claim 1, wherein the image fusion module unit is further configured to:

extracting landmark points of the CT image and landmark points of the MRI image after vertebrae segmentation;

Sequentially taking the points of landmark points of the CT image and the points of landmark points of the MRI image, wherein the number of the points is the same;

sequentially aligning the landmark points of the CT image with the landmark points of the MRI image in a mass center manner to obtain an initial alignment image;