CN115619790A - Mixed perspective method, system and equipment based on binocular positioning - Google Patents
Mixed perspective method, system and equipment based on binocular positioning Download PDFInfo
- Publication number
- CN115619790A CN115619790A CN202211636244.XA CN202211636244A CN115619790A CN 115619790 A CN115619790 A CN 115619790A CN 202211636244 A CN202211636244 A CN 202211636244A CN 115619790 A CN115619790 A CN 115619790A
- Authority
- CN
- China
- Prior art keywords
- reference frame
- image
- human body
- patient
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 210000000056 organ Anatomy 0.000 claims abstract description 63
- 238000013507 mapping Methods 0.000 claims abstract description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 11
- 230000004807 localization Effects 0.000 claims description 10
- 230000011218 segmentation Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 5
- 238000002594 fluoroscopy Methods 0.000 claims description 4
- 238000001356 surgical procedure Methods 0.000 claims description 4
- 238000010801 machine learning Methods 0.000 claims description 3
- 230000005764 inhibitory process Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 description 14
- 238000012800 visualization Methods 0.000 description 8
- 210000001519 tissue Anatomy 0.000 description 7
- 230000000007 visual effect Effects 0.000 description 7
- 238000003709 image segmentation Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 238000007781 pre-processing Methods 0.000 description 5
- 238000005070 sampling Methods 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 210000004204 blood vessel Anatomy 0.000 description 3
- 210000000988 bone and bone Anatomy 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003902 lesion Effects 0.000 description 3
- 210000005036 nerve Anatomy 0.000 description 3
- 230000001629 suppression Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003702 image correction Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000002604 ultrasonography Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 210000003484 anatomy Anatomy 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000010561 standard procedure Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/25—User interfaces for surgical systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/101—Computer-aided simulation of surgical operations
- A61B2034/102—Modelling of surgical devices, implants or prosthesis
- A61B2034/104—Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/101—Computer-aided simulation of surgical operations
- A61B2034/105—Modelling of the patient, e.g. for ligaments or bones
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/107—Visualisation of planned trajectories or target regions
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/108—Computer aided selection or customisation of medical implants or cutting guides
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
- A61B2034/2057—Details of tracking cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2065—Tracking using image or pattern recognition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2068—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis using pointers, e.g. pointers having reference marks for determining coordinates of body points
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10088—Magnetic resonance imaging [MRI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
The invention relates to a binocular positioning-based hybrid perspective method, system and equipment. The method comprises the following steps: obtaining a three-dimensional organ model of a patient; respectively acquiring a reference frame image and a patient body image based on the two cameras, constructing a reference frame mask image according to the reference frame image acquired in real time, and obtaining a patient body model through registration mapping; and based on the position relation among the three-dimensional organ model, the patient human body model, the reference frame mask image and the double-camera coordinate system, simultaneously projecting the three-dimensional organ model and the reference frame mask image on the human body model displayed in real time in the double-camera video to obtain a mixed perspective video image. The method aims to perform space positioning based on double cameras, and explores potential application value of the method in assisting a surgeon in achieving the best operation result by mixing a three-dimensional model of a real-time projection reference frame on a human body in a video and a three-dimensional reconstruction organ model.
Description
Technical Field
The invention relates to the field of image visualization analysis in clinical medicine, in particular to a binocular positioning-based hybrid perspective method, a binocular positioning-based hybrid perspective system, binocular positioning-based hybrid perspective equipment, a computer-readable storage medium and application of the computer-readable storage medium.
Background
The operation navigation is a visual image guide operation technology which is developed by taking medical images such as ultrasound, X-ray, CT, MRI and the like as basic data and by means of a computer, a precision instrument and image processing, can track the position of an operation instrument in real time through three-dimensional digital patient focus tissues, and realizes the visualization and automation of a surgical operation, thereby assisting a doctor or a robot to complete an operation task more quickly, accurately and safely.
Currently, doctors make diagnoses mainly by observing two-dimensional images such as ultrasound, X-ray, CT, and MRI. However, the two-dimensional image cannot visually display the three-dimensional anatomical structure of the focus region, and needs a doctor to deduce by experience; meanwhile, the problems of aliasing, noise, artifacts and the like exist in the image, and the accurate judgment of the state of an illness by a doctor is influenced. Thus, the success rate of the surgery depends greatly on the past experience of the doctor to perform the surgery, and the formation of the past experience requires a long period of medical training and clinical experience. In view of the fact that the three-dimensional model reconstruction is to establish the same three-dimensional virtual model in a computer for an objective object, the original data of two-dimensional medical images of a patient such as CT, MRI and the like can be imported into a three-dimensional model reconstruction system to form a three-dimensional visual digital model of the examined part of the patient.
Disclosure of Invention
An object of the present application is to provide a binocular positioning-based hybrid fluoroscopy method, system, device, computer-readable storage medium, and applications thereof, which are directed to positioning based on a binocular camera, solve the problem of navigation visualization through hybrid fluoroscopy, and superimpose a three-dimensional model of a surgical tool, a three-dimensional organ model of a patient, and a human body photographed by the camera during navigation, so as to find potential application values in assisting a surgeon in achieving an optimal surgical result, and provide more sufficient support for selection of a treatment decision.
According to a first aspect of the present application, an embodiment of the present application provides a binocular positioning-based hybrid perspective method, which includes:
obtaining a three-dimensional organ model of a patient;
acquiring a reference frame image based on the double cameras, and acquiring real-time spatial pose information of the reference frame in a double-camera coordinate system according to the reference frame image;
constructing a reference frame mask image based on the space pose information to obtain a real-time reference frame mask image;
acquiring a human body image of a patient based on the two cameras, determining a mapping relation between a human body coordinate system and the two-camera coordinate system through registration, and mapping to obtain a human body model of the patient;
based on the position relation of the three-dimensional organ model, the patient human body model, the real-time reference frame mask image and the coordinate system of the two cameras, the three-dimensional organ model and the real-time reference frame mask image are projected on the human body model displayed in real time in the video of the two cameras at the same time, and a mixed perspective video image is obtained.
In one embodiment, the three-dimensional organ model of the patient is based on medical images of various organs of the patient prior to surgery in a 1:1 ratio, and performing three-dimensional organ reconstruction.
Furthermore, the medical images of various organs comprise CT images and/or MRI images, and the three-dimensional organ reconstruction is to input the medical images of various organs before the operation of the patient into the trained human body three-dimensional organ model to obtain the three-dimensional organ model of the patient.
Optionally, the trained three-dimensional organ model of the human body is implemented by any one or more of the following network models: the multi-view three-dimensional reconstruction network pixelNeRF, MVSNet, patchMatchNet.
In an embodiment, the acquiring the reference frame image based on the dual cameras further comprises: the method comprises the steps of positioning the reference frame through binocular positioning, specifically, calibrating a left camera and a right camera and identifying binocular correction feature points through a Zhang Zhen friend calibration method, carrying out self-adaptive search frame radius based on black and white alternative feature points of the reference frame, detecting a candidate area which meets the requirements and only has the black and white alternative points along the edge of the search frame, carrying out symmetry detection on the candidate area, filtering the area which does not meet the requirements, carrying out convolution on the candidate area which meets the requirements to be used as an integral to generate an integral map, carrying out non-maximum value inhibition on the integral map and calculating the position of a sub-pixel point to determine the final feature point position, and positioning to obtain the key feature point of the reference frame.
In an embodiment, the real-time reference frame mask image is generated based on key feature points in the spatial pose information, specifically, whether a special shape specified by the reference frame exists or not is searched for by traversing feature points in left and right images acquired by two cameras, unpaired feature points are filtered out, a corresponding relation is detected by successfully matching the feature points, and the reference frame mask image is obtained by calculation through a least square method. Wherein the reference frame is a surgical tool.
Further, the spatial pose information includes key feature points of the frame of reference having a particular structure including a plurality of black-and-white alternating target regions of black-and-white alternating feature points.
In some embodiments, the acquiring a human body image of a patient based on two cameras further includes automatically segmenting and positioning a target region of the human body image of the patient by a machine learning method to obtain key position information and posture information of the target region, and further determining a human body coordinate system based on the key position information and posture information of the target region; optionally, the automatic segmentation and positioning are implemented by any one or more of the following algorithms: watershed segmentation, U-Net, MIScnn, resUNet.
In an embodiment, the registration determines a mapping relationship between a human body coordinate system and a dual-camera coordinate system by a point cloud registration method, wherein the point cloud registration is performed based on a mixed manner of global feature registration and local feature registration.
Further, the point cloud registration adopts any one or more of the following methods: 3Dsc, 4PCS, super4PCS, K-4PCS.
According to a second aspect of the present application, an embodiment of the present application provides a binocular positioning-based hybrid perspective system, which includes:
the three-dimensional model acquisition module is used for acquiring a three-dimensional organ model of a patient;
the coordinate system determination module is used for acquiring a reference frame image based on the double cameras and obtaining real-time space pose information of the reference frame in the double-camera coordinate system according to the reference frame image;
the mask image generation module is used for constructing a reference frame mask image based on the space pose information to obtain a real-time reference frame mask image;
the human body model registration module acquires a human body image of a patient based on the two cameras, determines the relation between a human body coordinate system and the two camera coordinate systems through registration, maps the human body coordinate system to the two camera coordinate systems based on the relation between the human body coordinate system and the two camera coordinate systems, and displays the human body model of the patient in real time in videos of the two cameras;
and the mixed perspective display module is used for simultaneously projecting the three-dimensional organ model and the real-time reference frame mask image on a human body model displayed in real time in the video of the two cameras, and obtaining a mixed perspective video image based on the position relation of the three-dimensional organ model, the human body model of the patient, the real-time reference frame mask image and the coordinate system of the two cameras.
According to a third aspect of the present application, an embodiment of the present application provides a binocular positioning-based hybrid perspective apparatus, which mainly includes:
a memory and a processor;
the memory is used for storing program instructions, the program instructions are used for storing a computer program of the mixed perspective based on the binocular positioning, and when the computer program is executed by the processor, the mixed perspective method based on the binocular positioning is realized;
the processor is configured to invoke program instructions that, when executed, perform a method for implementing a binocular localization-based hybrid perspective as described above.
According to a fourth aspect of the present application, an embodiment of the present application provides a computer-readable storage medium, on which a computer program of a binocular localization-based hybrid perspective is stored, which, when executed by a processor, implements the above binocular localization-based hybrid perspective method.
The application of the device or the system in intelligent navigation visualization of the operation; optionally, the visual application includes unifying the virtual coordinate system and the camera through a binocular calibration and registration algorithm, and projecting a three-dimensional model of the surgical tool and a three-dimensional reconstructed human organ model on the human body in real time in the video through a mixed perspective;
the application of the device or the system in extracting depth information; optionally, the applying includes extracting depth information from the image by spatial positioning when a plurality of cameras view the same scene;
the use of the above-described apparatus or system to assist a surgeon in performing accurate diagnostic analyses; optionally, the assisting includes: the reconstructed three-dimensional model can visually display the tissue structures of blood vessels, nerves, bones and the like in a focus area, can be rotated, zoomed and measured at will, is used for accurately positioning the focus position, determines the spatial adjacent relation between the focus and surrounding tissues and effectively assists doctors in accurate diagnosis.
The invention is based on the imaging principle and the binocular calibration technology, better reflects the specific situation and the effect of the operation tool, the patient human body model and the three-dimensional organ model through mixed perspective, is bedside, non-invasive, non-radiative and more practical, improves the visualization of navigation in the operation, has strong innovation, and has beneficial promoting effect on assisting a surgeon to realize the optimal operation result.
The application has the advantages that:
1. the application innovatively discloses a novel technology of mixed perspective based on binocular positioning, aiming at improving the visualization of navigation, the error precision of three-dimensional data and video superposition is within 1mm, and a three-dimensional model of a surgical tool and a three-dimensional reconstructed human organ model can be superposed, so that the navigation visual effect is more visual, the surgical difficulty is reduced, the surgical precision and success rate are improved, and the precision and depth of data analysis are objectively improved;
2. the three-dimensional model of the operation tool, the human body model of the patient and the three-dimensional organ model are innovatively integrated through the high-performance computer, the image data of the patient before the operation is connected with the specific position of the focus in the operation, the three-dimensional organ model can visually display the tissue structures of blood vessels, nerves, bones and the like in the focus area, can be rotated, zoomed and measured at will, is used for accurately positioning the position of the focus, effectively assists a doctor to make an optimal operation scheme, and is noninvasive, radiation-free and obvious in timeliness;
3. the application creatively discloses that a virtual coordinate system is unified with a camera through a binocular calibration and registration algorithm, and meanwhile, a three-dimensional model of an operation tool and a three-dimensional reconstructed human organ model are projected on a human body in a video in real time through a mixed perspective, so that the application is more accurately applied to the auxiliary analysis of the optimal operation result realized by a surgeon in view of the fact that the obtained mixed perspective video image has important research significance on operation navigation and intraoperative prevention and control.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a binocular positioning-based hybrid perspective method according to an embodiment of the present invention;
FIG. 2 is a flow chart of algorithm design for binocular positioning-based hybrid perspective display according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating a module composition of a hybrid perspective system based on binocular positioning according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a hybrid perspective device based on binocular positioning according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.
In some of the flows described in the present specification and claims and in the above-described figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, with the order of the operations, e.g., S101, S102, etc., merely being used to distinguish between various operations, and the order of the operations itself does not represent any order of performance. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the application provides a mixed perspective method based on binocular positioning, a mixed perspective system based on binocular positioning, mixed perspective equipment based on binocular positioning and a computer readable storage medium. The binocular positioning-based hybrid perspective equipment comprises a terminal or a server and other equipment. The terminal can be terminal equipment such as a smart phone, a tablet computer, a notebook computer and a personal computer. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content Delivery Network (CDN) and a big data and artificial intelligence platform. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein.
Fig. 1 is a schematic flow chart of a binocular positioning-based hybrid perspective method provided in an embodiment of the present invention, specifically, the method includes the following steps:
s101: a three-dimensional organ model of a patient is obtained.
In one embodiment, a three-dimensional organ model of a patient is based on medical images of various organs prior to a patient in a 1:1 ratio, and performing three-dimensional organ reconstruction. The medical images of various organs before the operation of the patient comprise CT images and MRI images.
Further, the three-dimensional organ reconstruction is to input medical images of various organs of the patient before the operation into the trained human three-dimensional organ model to obtain the three-dimensional organ model of the patient.
Optionally, the trained three-dimensional organ model of the human body is implemented by any one or more of the following network models: pixelNeRF, MVSNet, patchMatchNet.
pixelNeRF is a NeRF-based multi-view three-dimensional reconstruction network that can be trained on top of a dataset of multi-view images, taking as input the aerial image features aligned to each pixel, predicting the NeRF representation in the camera coordinate system of the input image, i.e. a three-dimensional reconstruction centered on the viewer.
MVSNet, based on a depth estimation network of a multi-view image, firstly performs feature extraction on a 2D image to obtain a feature map, then constructs a 3D cost body based on a camera view cone of a reference view through differentiable homography transformation, performs regularization by using 3D convolution, and regresses to obtain an initial depth map and a final depth map.
PatchMatchNet, an efficient multi-view stereo framework, reconstructs scenes as point clouds or mesh given some images and corresponding camera parameters (including internal and external parameters).
In one specific embodiment, the method for obtaining the trained three-dimensional organ model of the human body comprises the following steps: performing adaptive adjustment and optimization on an original network model based on medical image data characteristics of various preoperative organs of the patient to obtain the preoperative medical image data; optionally, the adaptive adjustment includes using a batch-standardized acceleration network convergence rate, an activation function, and a Dice Loss function optimization model.
In one embodiment, the reconstruction of three-dimensional organ models can be classified into three categories according to the imaging principles: and (3) three-dimensional reconstruction based on CT images and MRI images.
In a specific embodiment, the three-dimensional organ model reconstructed based on the preoperative CT image and/or MRI image of the patient can visually display the tissue structures of blood vessels, nerves, bones and the like in the lesion area, can be rotated, zoomed and measured at will, is used for accurately positioning the lesion position, defines the spatial adjacent relation between the lesion and surrounding tissues, and effectively assists a doctor in accurate diagnosis.
S102: and acquiring a reference frame image based on the two cameras, and acquiring real-time spatial pose information of the reference frame in a coordinate system of the two cameras according to the reference frame image.
In one embodiment, the reference frame image acquisition based on the double cameras comprises preprocessing of adaptive binarization processing, gaussian filtering and the like on the acquired reference frame image.
In a specific embodiment, acquiring the reference frame image based on the dual cameras further comprises: and positioning the reference frame through binocular positioning. Specifically, binocular positioning is performed by calibrating a left camera and a right camera and identifying binocular correction feature points by using a Zhang Zhengyou calibration method, self-adaptive search frame radius is performed based on black and white alternate feature points of a reference frame, candidate areas which meet conditions and only have black and white alternate points are detected along the edge of the search frame, symmetry detection is performed on the candidate areas, areas which do not meet the conditions are filtered, the candidate areas which meet the conditions are convolved to be used as integrals to generate integral graphs, the positions of the final feature points are determined by performing non-maximum suppression and sub-pixel point position calculation on the integral graphs, and key feature points of the reference frame are obtained by positioning.
Specifically, the calibration method of the left camera and the right camera aims at estimating parameters forming a camera calibration matrix, and a calibration process adopts a Sudoku calibration technology of Zhangnyang, constructs a plurality of sub-formulas and solves the parameters. When multiple cameras view the same scene, depth information can be extracted from the image.
In one embodiment, the real-time reference frame mask image is generated based on key feature points in the spatial pose information, specifically, whether a special shape specified by the reference frame exists or not is searched for by traversing feature points in the left and right images acquired by the double cameras, unpaired feature points are filtered out, the corresponding relation is detected by successfully matching the feature points, and the reference frame mask image is obtained by calculation through a least square method. Wherein, the reference frame refers to a surgical tool.
Further, the spatial pose information comprises key feature points of the reference frame with a specific structure, and the key features of the specific structure comprise a plurality of black-and-white alternating target areas formed by black-and-white alternating feature points.
S103: and constructing a reference frame mask image based on the space pose information to obtain a real-time reference frame mask image.
In one embodiment, the constructed mask image of the reference frame is three-dimensionally reconstructed based on the spatial pose information of the real-time reference frame in the dual-camera coordinate system obtained in step S102, and the obtained mask image of the real-time reference frame is obtained based on the real-existing reference frame according to the following ratio of 1:1 ratio of the reconstructed three-dimensional image.
Specifically, the method for performing three-dimensional reconstruction is implemented by using any one or more of the network models in step S101: pixelNeRF, MVSNet, patchMatchNet, which are not described herein.
S104: the method comprises the steps of obtaining a human body image of a patient based on two cameras, determining a mapping relation between a human body coordinate system and the two camera coordinate systems through registration, and mapping to obtain a human body model of the patient.
In an embodiment, based on a human body image of a patient acquired by two cameras, a mapping relation between a human body coordinate system and two camera coordinate systems is determined by a mixed mode of global feature registration and local feature registration in a point cloud registration method, and the image data of the patient in the two camera coordinate systems is obtained according to the following formula 1:1 scale reproduction of the patient phantom.
Further, the point cloud registration adopts any one or more of the following methods: 3Dsc, 4PCS, super4PCS, K-4PCS.
The 3Dsc is a point cloud registration method based on a point cloud descriptor consisting of a Local Reference Frame (LRF) of 3D, and rotation invariance is realized by calculating interest points and aligning the interest points with the LRF.
4PCS (four-point process) registration algorithm, namely a four-point method registration algorithm, constructs a spatial topological relation between the target point cloud and the matching points according to the affine invariance of non-coplanar four points in the original point cloud, matches corresponding point pairs meeting conditions in a coplanar four-point set, uses LCP (Large Common Pointset) strategy to find the four-point pair with the maximum overlapping degree after registration to obtain an optimal matching result, and completes the rough matching of the point cloud.
super4PCS, improved based on 4PCS registration algorithm, accelerates the search process of coplanar four-point set by using rasterized point cloud, and remarkably reduces the computational complexity of 4PCS algorithm based on intelligent index.
The K-4PCS utilizes a VoxelGrid filter to carry out down-sampling on the point cloud Q, then uses a standard method (3D harris or 3D DoG) to carry out 3D key point detection, and uses a key point set instead of an original point cloud to carry out data matching through a 4PCS algorithm, so that the scale of a search point set is reduced, and the operation efficiency is improved.
In a specific embodiment, a mapping relation between a human body coordinate system and a double-camera coordinate system is determined by combining a registration method of 3Dsc and super4pcs based on a human body image of a patient acquired by double cameras, and a human body model of the patient based on the double-camera coordinate system is obtained through mapping.
In some embodiments, the method includes acquiring a human body image of a patient based on two cameras, automatically segmenting and positioning a target region of the acquired human body image of the patient by a machine learning method to obtain key position information and posture information of the target region, and determining a human body coordinate system based on the key position information and posture information of the target region.
Further, optionally, the automatic segmentation and localization are implemented by any one or several of the following algorithms: watershed segmentation, U-Net, MIScnn, swin-Unet, UTNet.
The watershed algorithm is a typical image segmentation algorithm based on edges, and the image segmentation method can be better suitable for target segmentation under a complex background by searching for boundaries between regions and segmenting the image, particularly content segmentation of a picture with a honeycomb structure.
The U-Net algorithm is a network model suitable for medical image segmentation, and Conv + Pooling down sampling is firstly carried out; then Deconv deconvolution is carried out for up-sampling, and low-layer feature maps before crop are fused; then up-sampling is carried out again, and the steps are repeated until an output target feature map is obtained, and finally a segmentation image is obtained through softmax.
MIScnn has a medical image segmentation framework of a convolutional neural network and deep learning, provides an intuitive and quick API for establishing a medical image segmentation process, and comprises data I/O, preprocessing, data enhancement, block-by-block analysis, evaluation indexes, a library with a latest deep learning model and model use.
Swin-Unet builds a symmetric coder-decoder architecture with jump connection based on Swin transform block, develops a patch extension layer, can realize up-sampling and increase of characteristic dimension without convolution or interpolation operation, and finally builds a U-shaped coding and decoding structure purely based on transform.
UTNet, a U-hybrid transformation network (UTNet), is used to integrate the advantages of convolutional layers and self-attention mechanisms for medical image segmentation.
S105: based on the position relation among the three-dimensional organ model, the patient human body model, the reference frame mask image and the double-camera coordinate system, the three-dimensional organ model and the reference frame mask image are projected on the human body model displayed in real time in the video of the double cameras at the same time, and a mixed perspective video image is obtained.
In one embodiment, the three-dimensional organ model, the patient human body model and the reference frame mask map constructed in the above steps are established through corresponding position relations based on a double-camera coordinate system, so that the three-dimensional organ model and the reference frame mask map are projected on the human body model displayed in real time in a video of double cameras simultaneously, and further a mixed perspective video image is obtained.
In a specific embodiment, the three-dimensional organ model and the human body shot by the cameras can be overlaid together in real time displayed in the video of the two cameras, and a reference frame mask image (namely, a three-dimensional model of a surgical tool) obtained in real time through a reference frame can also be overlaid with the three-dimensional reconstructed human body organ model, so that the visualization of surgical navigation is obviously improved, the navigation visual effect is more visual, and the surgical difficulty is obviously reduced.
In a more complete embodiment, the method shown in fig. 1 is applied to implement a binocular positioning-based hybrid perspective process by an algorithm design as shown in fig. 2. In the mixed perspective process based on binocular positioning shown in fig. 2, firstly, camera calibration is performed, then, image acquisition, image preprocessing, feature Point recognition (black and white alternating X-Point), positioning tool matching and human body model matching are performed in sequence, and then, a mixed perspective video image is obtained through mixed perspective display.
Specifically, the camera calibration process: calibrating a single camera in the left camera and the right camera based on a Zhangyingyou calibration method, analyzing calibration errors by measuring internal and external parameters and distortion coefficients of the cameras, calibrating and correcting the binocular based on an OpenCV (open content computer vision) general algorithm to determine the relative pose of the binocular when the errors are lower than a threshold value, analyzing the calibration and correction errors, and performing the next step, namely acquiring images of the double cameras when the calibration and correction errors reach the set threshold value.
Further, double-camera image acquisition is carried out based on a background thread and cache mode, and acquisition is realized through image correction (lmage retrieval) to obtain a reference frame image. The image correction uses the image central point as a rotation center, and counterclockwise rotation is carried out on the image by using the image rotation angle as a reference, so that the image realizes inclination correction; specifically, the method comprises the following steps of obtaining the position and rotation information of the area to be corrected:
(1) Acquiring position information of an area to be corrected, mainly acquiring coordinates of a central point of an external rectangle outside the area to be corrected and coordinates of four vertexes of the central point, and facilitating coordinate migration;
(2) Acquiring the rotation angle of the region to be corrected: and taking the coordinate position of the center of the rectangle outside the image to be corrected as the origin of coordinates to serve as a plane rectangular coordinate system, wherein the included angle between the long side of the rectangle outside the image to be corrected and the plane rectangular coordinate system is the rotation angle of the image to be corrected.
The image preprocessing process mainly comprises the steps of carrying out a plurality of operations such as adaptive binarization processing, gaussian filtering, threshold segmentation, contour searching and the like on the obtained reference frame image to obtain a preprocessed reference frame image.
Further, feature Point identification (black and white alternating X-Point) mainly performs key feature identification on the preprocessed reference frame image to obtain real-time spatial pose information of the reference frame in a double-camera coordinate system. The specific identification process is as follows:
firstly, carrying out self-adaptive search box radius based on a preprocessed reference frame image;
then, detecting key feature points satisfying the specific structure of the reference frame along the edge of the search frame, for example, searching whether a candidate region satisfying and having only 4 black-white alternating points, that is, a plurality of black-white alternating target regions composed of black-white alternating feature points;
then, symmetry detection is carried out on the candidate regions, and regions which do not meet the conditions are filtered. The symmetry detection mainly judges whether the image is a black-white alternating region and the proportion of the black area to the white area, specifically, binocular calculation is carried out on the feature points matched with the left image and the right image based on the spatial position calculation of the feature points, and the feature points are traversed to find whether a specified special shape (such as a black-white alternating symmetric region) exists or not so as to match the symmetry detection;
then, performing convolution on the candidate area meeting the condition to be used as an integral (BoxFilter) to generate an integral graph;
then, performing non-maximum suppression and sub-pixel point position calculation on the generated integral image to determine the final characteristic point position;
and finally, performing feature point matching on the left image and the right image, filtering out unpaired feature points, further obtaining real-time space pose information of the reference frame on the double-camera coordinate system, and positioning to obtain key feature points of the reference frame.
Further, the positioning tool matching process mainly searches whether a specific special shape exists by traversing the feature points to match a specific reference frame (i.e. the positioning tool). For example, for a frame of reference consisting of 4 feature points, 4 special triangles may be produced, at which time it is looked up whether there is a specific special triangle to match the frame of reference. Specifically, whether a specified special triangle exists or not is searched for through the real-time spatial position and attitude information (feature points) of the reference frame in the double-camera coordinate system obtained through traversal, unpaired feature points are filtered out, the corresponding relation is detected through the feature points which are successfully paired, and a reference frame mask image is obtained through calculation by using a least square method.
Further, the step of registering the human body model can determine the relation between the human body coordinate system and the camera coordinate system through four-point registration, so that the human body model of the patient under the double-camera coordinate system is obtained.
And further, performing mixed perspective display, namely mapping the patient human body model to the double-camera coordinate system through the relation between the human body model coordinate system and the camera coordinate system, and displaying the patient human body model in real time in the video of the double cameras. Specifically, based on the position relationship among the three-dimensional organ model, the patient human body model, the reference frame mask image and the coordinate system of the two cameras, the three-dimensional organ model and the real-time reference frame mask image are projected on the human body model displayed in real time in the video of the two cameras at the same time, and therefore the mixed perspective video image is obtained.
In one embodiment, since real-time computation is required and spatial positioning requires at least 3 points, when a reference frame is designed as a tool with four irregular points coplanar, in order to accurately position, a circle is drawn with the point as the center, and the circle 4 is equally divided into sector areas alternating black and white, then a binocular positioning-based hybrid perspective process can be described as: firstly, calibrating and binocular correcting a left camera and a right camera by using a Zhang friend calibration method; further, according to the calibration and binocular correction results, the image acquisition and preprocessing process is expanded as follows: the images generated by the left camera and the right camera are uploaded to a GPU for accelerated calculation, meanwhile, the color images are converted into gray images, and Gaussian noise reduction is carried out; and then, carrying out area binarization: the picture is divided into a plurality of appropriate square areas, and the mean value of each area is taken as a threshold value to carry out binarization to obtain a binarization picture, so that the influence of the illumination environment is reduced, and local features are highlighted.
Further, the target area is rapidly identified by the candidate feature points based on the calculation of the binary image, specifically:
(1) And selecting a proper characteristic point identification area radius.
(2) A circle is drawn at a specified radius for each pixel point p0.
(3) And calculating the gradient value of each pixel point on the circumference along the clockwise direction.
(4) Counting the number and coordinates of pixel points (pts) with gradient values of 255 on the circumference, and screening a point p0 with the pts meeting the requirement that the number is 4 and the sum of the 4 gradient values is close to 0.
(5) The total of 5 points p0 and pts form 4 triangular regions.
(6) And respectively carrying out a random point-taking algorithm on the 4 triangular areas to obtain a proper amount of pixel values so as to obtain color values of the 4 areas.
(7) And screening points which are two black and two white and are alternately distributed in 4 areas.
(8) And carrying out triangle congruent judgment on every two areas with the same color in the 4 triangle areas.
(9) And finally, removing the points which do not meet the conditions by utilizing the characteristic that the same characteristic points have the same vertical coordinate in the left graph and the right graph after binocular stereo correction, and recording the obtained proper characteristic points as candidate points p1.
Still further, calculating the sub-pixel positions of the feature points by using the gray map:
(1) Drawing 4 line segments with the included angles of 0 degree, 45 degrees, 90 degrees and 135 degrees with the horizontal direction respectively at the candidate point P1 as the center, and accumulating the values of pixel points through which the line segments pass to obtain 4 accumulated values. Then, the maximum value minus the minimum value is used as the integral of the candidate points to generate an integral graph.
(2) And (4) performing a maximum level value suppression algorithm on the integral graph to obtain the feature point coordinate with the highest integral in the area, and selecting the feature point coordinate as the last correct feature point.
(3) And calculating horizontal and vertical offset values according to the integral distribution conditions around the feature points to obtain the calculation of the spatial position of the sub-pixel coordinate feature points of the feature points. More specifically, the spatial three-dimensional space coordinates of the feature points are calculated by using a parallax method.
And further, matching and positioning the characteristic points and the multiplex tools:
(1) Among the feature points used, a specific triangle is looked up to match the one-to-one correspondence of different tools and points.
(2) And calculating the spatial pose of the tool by using a least square method according to the corresponding relation of the 4 points.
Then, based on the spatial positioning of the tool under the two cameras, a coordinate system is established through a registration algorithm, so that the virtual coordinate system is unified with the cameras, and the projected three-dimensional tool mask map is displayed on the human body in the video at the same time; meanwhile, a three-dimensional organ model reconstructed based on image data before the operation of a patient is connected with the specific position of a focus in the operation through a high-performance computer, and the three-dimensional space position of the focus and adjacent important tissue organs are accurately displayed, so that a doctor can select the optimal operation access through real-time navigation information to make or implement an optimal operation scheme.
The method is feasible to be used for operation navigation visualization, and the same principle shows that the error precision of the three-dimensional data and the video superposition is within 1mm, so that the aim of greatly reducing the operation navigation cost is fulfilled on the premise of meeting the precision requirement and stability requirement of clinical use. The method better reflects the specific combination condition and effect of the operation tool, the patient human body model and the three-dimensional organ model through mixed perspective, makes full use of more accurate real-time navigation information to enable the patient to obtain safe, accurate and minimally invasive operation treatment, is a bedside, non-invasive, non-radiative and more practical method, and is more beneficial to assisting a surgeon in realizing auxiliary diagnosis in the aspect of the optimal operation result and prevention and control analysis of occurrence and development of related diseases.
The embodiment of the invention provides a binocular positioning-based hybrid perspective system, as shown in fig. 3, which comprises:
s301: and a three-dimensional model acquisition module. The module is used to acquire a three-dimensional organ model of a patient.
S302: a coordinate system determination module. The module obtains real-time space pose information of the reference frame in a double-camera coordinate system based on the reference frame images acquired by the double cameras.
S303: and a mask image generation module. The module constructs a reference frame mask image based on the space pose information obtained in the step S302 to obtain a real-time reference frame mask image.
S304: and a human body model registration module. The module determines the mapping relation between a human body coordinate system and two camera coordinate systems through registration based on a human body image of a patient acquired by two cameras, maps the human body coordinate system to the lower part of the two camera coordinate systems based on the determined relation between the human body coordinate system and the two camera coordinate systems, and displays the human body model of the patient in real time in a video of the two cameras.
S305: a hybrid see-through display module. The module is used for simultaneously projecting the three-dimensional organ model obtained in the step S301 and the reference frame mask image obtained in the step S303 on the human body model displayed in real time in the video of the double cameras, and specifically, the mixed perspective video image is obtained based on the position relation of the three-dimensional organ model, the human body model of the patient, the reference frame mask image and the coordinate system of the double cameras.
Fig. 4 is a hybrid perspective device based on binocular positioning according to an embodiment of the present invention, which includes:
a memory and a processor;
the apparatus may further include: an input device and an output device.
The memory, the processor, the input device and the output device may be connected by a bus or other means, and the bus connection is illustrated in fig. 4 as an example; the memory is used for storing program instructions, the stored program instructions are computer programs of mixed perspective based on binocular positioning, and when the computer programs are executed by the processor, the mixed perspective method based on binocular positioning is realized; the processor is configured to invoke program instructions that, when executed, perform a method of implementing binocular localization based hybrid perspective as described above.
The invention provides a computer readable storage medium, on which a computer program is stored, the computer program, when executed by a processor, implements the binocular positioning-based hybrid perspective method.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the apparatus, or the module described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative; for example, the division of the module is only a logic function division, and there may be another division manner in actual implementation; as another example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
Modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Specifically, some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like.
It will be understood by those skilled in the art that all or part of the steps in the method according to the above embodiments may be implemented by hardware that is instructed by a program, and the program may be stored in a computer-readable storage medium, such as a read-only memory, a magnetic or optical disk, and the like.
While the invention has been described in detail with reference to certain embodiments, it will be apparent to one skilled in the art that the invention may be practiced without these specific details.
Claims (10)
1. A binocular positioning-based hybrid perspective method comprises the following steps:
obtaining a three-dimensional organ model of a patient;
acquiring a reference frame image based on the double cameras, and acquiring real-time spatial pose information of the reference frame in a double-camera coordinate system according to the reference frame image;
constructing a reference frame mask image based on the space pose information to obtain a real-time reference frame mask image;
acquiring a human body image of a patient based on the two cameras, determining a mapping relation between a human body coordinate system and the two-camera coordinate system through registration, and mapping to obtain a human body model of the patient;
based on the position relation of the three-dimensional organ model, the patient human body model, the real-time reference frame mask image and the coordinate system of the double cameras, the three-dimensional organ model and the real-time reference frame mask image are projected on the human body model displayed in real time in the video of the double cameras at the same time, and a mixed perspective video image is obtained.
2. The binocular positioning-based hybrid fluoroscopy method as claimed in claim 1, wherein the three-dimensional organ model of the patient is based on medical images of various organs of the patient before surgery in a ratio of 1:1 proportion to carry out three-dimensional organ reconstruction.
3. The binocular positioning-based hybrid perspective method of claim 1, wherein the spatial pose information comprises key feature points of the frame of reference having a specific structure comprising a plurality of black-and-white alternating target regions of black-and-white alternating feature points.
4. The binocular positioning-based hybrid perspective method of claim 1, wherein the acquiring the frame of reference images based on the two cameras further comprises: the method comprises the steps of positioning the reference frame through binocular positioning, specifically, calibrating a left camera and a right camera and identifying binocular correction feature points through a Zhang Zhen friend calibration method, carrying out self-adaptive search frame radius based on black and white alternative feature points of the reference frame, detecting a candidate area which meets the requirements and only has the black and white alternative points along the edge of the search frame, carrying out symmetry detection on the candidate area, filtering the area which does not meet the requirements, carrying out convolution on the candidate area which meets the requirements to be used as an integral to generate an integral map, carrying out non-maximum value inhibition on the integral map and calculating the position of a sub-pixel point to determine the final feature point position, and positioning to obtain the key feature point of the reference frame.
5. The binocular positioning-based hybrid perspective method according to claim 1, wherein the real-time reference frame mask image is generated based on key feature points in the spatial pose information, specifically, whether a special shape designated by the reference frame exists or not is found by traversing feature points in left and right images acquired by two cameras, unmatched feature points are filtered, a corresponding relationship is detected by successfully matching the feature points, and the reference frame mask image is obtained by calculation with a least square method.
6. The binocular positioning-based hybrid perspective method according to claim 1, wherein the patient body image is acquired based on two cameras, the method further comprises the steps of automatically segmenting and positioning a target region of the patient body image by a machine learning method to obtain key position information and posture information of the target region, and further determining a body coordinate system based on the key position information and the posture information of the target region; optionally, the automatic segmentation and positioning are implemented by any one or more of the following algorithms: watershed segmentation, U-Net, MIScnn, resUNet.
7. The binocular localization-based hybrid perspective method according to claim 1, wherein the registration is performed by determining a mapping relationship between a human coordinate system and a dual-camera coordinate system by a point cloud registration method, and the point cloud registration is performed based on a hybrid manner of global feature registration and local feature registration.
8. The binocular localization-based hybrid perspective method of claim 7, wherein the point cloud registration employs one or more of the following methods: 3Dsc, 4PCS, super4PCS, K-4PCS.
9. A binocular positioning-based hybrid perspective system, comprising:
the three-dimensional model acquisition module is used for acquiring a three-dimensional organ model of a patient;
the coordinate system determination module is used for acquiring a reference frame image based on the double cameras and obtaining real-time space pose information of the reference frame in the double-camera coordinate system according to the reference frame image;
the mask image generation module is used for constructing a reference frame mask image based on the space pose information to obtain a real-time reference frame mask image;
the human body model registration module is used for acquiring a human body image of a patient based on the two cameras, determining the relation between a human body coordinate system and the two camera coordinate systems through registration, mapping the human body coordinate system to the two camera coordinate systems based on the relation between the human body coordinate system and the two camera coordinate systems, and displaying the human body model of the patient in real time in videos of the two cameras;
and the mixed perspective display module is used for simultaneously projecting the three-dimensional organ model and the real-time reference frame mask image on a human body model displayed in real time in the video of the two cameras, and obtaining a mixed perspective video image based on the position relation of the three-dimensional organ model, the human body model of the patient, the real-time reference frame mask image and the coordinate system of the two cameras.
10. A binocular positioning-based hybrid fluoroscopy apparatus, the apparatus comprising:
a memory and a processor;
the memory for storing program instructions for storing a computer program of binocular localization based hybrid perspective, which when executed by the processor implements a binocular localization based hybrid perspective method of any one of claims 1-8;
the processor is configured to invoke program instructions that, when executed, perform a method of implementing a binocular localization-based hybrid perspective of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211636244.XA CN115619790B (en) | 2022-12-20 | 2022-12-20 | Hybrid perspective method, system and equipment based on binocular positioning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211636244.XA CN115619790B (en) | 2022-12-20 | 2022-12-20 | Hybrid perspective method, system and equipment based on binocular positioning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115619790A true CN115619790A (en) | 2023-01-17 |
| CN115619790B CN115619790B (en) | 2023-05-02 |
Family
ID=84880980
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211636244.XA Active CN115619790B (en) | 2022-12-20 | 2022-12-20 | Hybrid perspective method, system and equipment based on binocular positioning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115619790B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116958265A (en) * | 2023-09-19 | 2023-10-27 | 交通运输部天津水运工程科学研究所 | Ship pose measurement method and system based on binocular vision |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020082498A1 (en) * | 2000-10-05 | 2002-06-27 | Siemens Corporate Research, Inc. | Intra-operative image-guided neurosurgery with augmented reality visualization |
| CN101073528A (en) * | 2007-06-22 | 2007-11-21 | 北京航空航天大学 | Digital operating bed system with double-plane positioning and double-eyes visual tracting |
| CN102784003A (en) * | 2012-07-20 | 2012-11-21 | 北京先临华宁医疗科技有限公司 | Pediculus arcus vertebrae internal fixation operation navigation system based on structured light scanning |
| CN103519895A (en) * | 2013-10-18 | 2014-01-22 | 江苏艾迪尔医疗科技股份有限公司 | Orthopedic operation auxiliary guide method |
| CN104586505A (en) * | 2014-09-19 | 2015-05-06 | 张巍 | Navigating system and method for orthopedic operation |
| CN109223121A (en) * | 2018-07-31 | 2019-01-18 | 广州狄卡视觉科技有限公司 | Based on medical image Model Reconstruction, the cerebral hemorrhage puncturing operation navigation system of positioning |
| CN109758230A (en) * | 2019-02-26 | 2019-05-17 | 中国电子科技集团公司信息科学研究院 | A kind of neurosurgery air navigation aid and system based on augmented reality |
| CN112168346A (en) * | 2020-11-02 | 2021-01-05 | 沈阳蓝软智能医疗科技有限公司 | Method for real-time coincidence of three-dimensional medical image and patient and operation auxiliary system |
| CN113499166A (en) * | 2021-06-21 | 2021-10-15 | 西安交通大学 | Autonomous stereoscopic vision navigation method and system for corneal transplantation surgical robot |
| CN114041878A (en) * | 2021-10-19 | 2022-02-15 | 山东建筑大学 | Three-dimensional reconstruction method and system for CT image of bone joint replacement surgical robot |
-
2022
- 2022-12-20 CN CN202211636244.XA patent/CN115619790B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020082498A1 (en) * | 2000-10-05 | 2002-06-27 | Siemens Corporate Research, Inc. | Intra-operative image-guided neurosurgery with augmented reality visualization |
| CN101073528A (en) * | 2007-06-22 | 2007-11-21 | 北京航空航天大学 | Digital operating bed system with double-plane positioning and double-eyes visual tracting |
| CN102784003A (en) * | 2012-07-20 | 2012-11-21 | 北京先临华宁医疗科技有限公司 | Pediculus arcus vertebrae internal fixation operation navigation system based on structured light scanning |
| CN103519895A (en) * | 2013-10-18 | 2014-01-22 | 江苏艾迪尔医疗科技股份有限公司 | Orthopedic operation auxiliary guide method |
| CN104586505A (en) * | 2014-09-19 | 2015-05-06 | 张巍 | Navigating system and method for orthopedic operation |
| CN109223121A (en) * | 2018-07-31 | 2019-01-18 | 广州狄卡视觉科技有限公司 | Based on medical image Model Reconstruction, the cerebral hemorrhage puncturing operation navigation system of positioning |
| CN109758230A (en) * | 2019-02-26 | 2019-05-17 | 中国电子科技集团公司信息科学研究院 | A kind of neurosurgery air navigation aid and system based on augmented reality |
| CN112168346A (en) * | 2020-11-02 | 2021-01-05 | 沈阳蓝软智能医疗科技有限公司 | Method for real-time coincidence of three-dimensional medical image and patient and operation auxiliary system |
| CN113499166A (en) * | 2021-06-21 | 2021-10-15 | 西安交通大学 | Autonomous stereoscopic vision navigation method and system for corneal transplantation surgical robot |
| CN114041878A (en) * | 2021-10-19 | 2022-02-15 | 山东建筑大学 | Three-dimensional reconstruction method and system for CT image of bone joint replacement surgical robot |
Non-Patent Citations (1)
| Title |
|---|
| 郭贝: "基于双目视觉的手术器械定位技术研究" * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116958265A (en) * | 2023-09-19 | 2023-10-27 | 交通运输部天津水运工程科学研究所 | Ship pose measurement method and system based on binocular vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115619790B (en) | 2023-05-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110010249B (en) | Augmented reality operation navigation method and system based on video superposition and electronic equipment | |
| US10674936B2 (en) | Hybrid registration method | |
| Lin et al. | Video‐based 3D reconstruction, laparoscope localization and deformation recovery for abdominal minimally invasive surgery: a survey | |
| Stoyanov | Stereoscopic scene flow for robotic assisted minimally invasive surgery | |
| Varnavas et al. | Fully automated 2D–3D registration and verification | |
| CN112150524B (en) | Two-dimensional and three-dimensional medical image registration method and system based on deep learning | |
| CN111640100A (en) | Tumor image processing method and device, electronic equipment and storage medium | |
| CN110288653B (en) | Multi-angle ultrasonic image fusion method and system and electronic equipment | |
| CN112907642B (en) | Registration and superposition method, system, storage medium and equipment | |
| Bernhardt et al. | Robust dense endoscopic stereo reconstruction for minimally invasive surgery | |
| WO2010081094A2 (en) | A system for registration and information overlay on deformable surfaces from video data | |
| US20230316639A1 (en) | Systems and methods for enhancing medical images | |
| CN111508068B (en) | Three-dimensional reconstruction method and system applied to binocular endoscopic image | |
| Patias | Medical imaging challenges photogrammetry | |
| CN115619790B (en) | Hybrid perspective method, system and equipment based on binocular positioning | |
| Kumar et al. | Stereoscopic visualization of laparoscope image using depth information from 3D model | |
| CN110916707A (en) | Two-dimensional bone image acquisition method, system and device | |
| Zhou et al. | Circular generalized cylinder fitting for 3D reconstruction in endoscopic imaging based on MRF | |
| CN113256754B (en) | Stacking projection reconstruction method for segmented small-area tumor mass | |
| CN114399527A (en) | Method and device for unsupervised depth and motion estimation of monocular endoscope | |
| CN114617527A (en) | Laparoscope three-dimensional imaging method and system | |
| CN113658106A (en) | Liver focus automatic diagnosis system based on abdomen enhanced CT | |
| CN111743628A (en) | Automatic puncture mechanical arm path planning method based on computer vision | |
| Amir-Khalili et al. | 3D surface reconstruction of organs using patient-specific shape priors in robot-assisted laparoscopic surgery | |
| EP4254350A1 (en) | Determination of illumination parameters in medical image rendering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 100082, 9th Floor, Building 1, No. 32 Xizhimen North Street, Haidian District, Beijing 1004A23 Patentee after: Beijing Weizhuo Zhiyuan Medical Technology Co.,Ltd. Country or region after: China Address before: 901, Building 4, Area 4, Hanwei International Plaza, Fengtai District, Beijing, 100071 Patentee before: VISUAL3D MEDICAL TECHNOLOGY DEVELOPMENT Co.,Ltd. Country or region before: China |