CN114092643A - Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN - Google Patents

Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN Download PDF

Info

Publication number
CN114092643A
CN114092643A CN202111372468.XA CN202111372468A CN114092643A CN 114092643 A CN114092643 A CN 114092643A CN 202111372468 A CN202111372468 A CN 202111372468A CN 114092643 A CN114092643 A CN 114092643A
Authority
CN
China
Prior art keywords
network
global
local
point cloud
deformation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111372468.XA
Other languages
Chinese (zh)
Inventor
夏羽
刘安然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen University
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to CN202111372468.XA priority Critical patent/CN114092643A/en
Publication of CN114092643A publication Critical patent/CN114092643A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Computer Hardware Design (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Epidemiology (AREA)
  • Medical Informatics (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a soft tissue self-adaptive deformation method and system based on mixed reality and 3DGAN, and belongs to the technical field of medical treatment. The method comprises the following steps: acquiring a preoperative global three-dimensional model and converting the preoperative global three-dimensional model into point cloud data; acquiring the local image depth map data of a patient in operation in real time, and converting the local image depth map data into point cloud data; sending the acquired point cloud data into a nested 3DGAN network model to realize local deformation mapping to global deformation; and converting the obtained point cloud global data after deformation into a three-dimensional model. The invention also provides a system for realizing the soft tissue self-adaptive deformation method based on the mixed reality and the 3 DGAN. The beneficial effects of the invention are as follows: the method can enable the deformation of human body soft tissue in the operation to be matched with a virtual three-dimensional model before the operation according to the local image of the patient in the operation process, and generate the three-dimensional model after the deformation of the soft tissue in the patient body in real time.

Description

Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN
Technical Field
The invention relates to a medical technology, in particular to a soft tissue self-adaptive deformation method based on mixed reality and 3 DGAN.
Background
In the traditional operation process, a doctor needs to combine professional knowledge and CT or MR scanning two-dimensional images before the operation of a patient to conceive a three-dimensional model in the body of the patient, and the experience of the doctor and the energy consumption of long-time high-load operation are very tested. The intraoperative navigation system is a novel operation auxiliary mode, a three-dimensional body model of a patient is matched with an intraoperative anatomical structure through a registration method, and then the virtual three-dimensional model is displayed to a doctor through corresponding imaging equipment after being overlapped with a real patient, so that the condition of the doctor on the anatomical position of the intraoperative patient is clear, the safety and the success rate of an operation are improved, and the operation burden of the doctor is effectively reduced.
The existing intraoperative navigation method has a plurality of defects, and mainly solves the problems that the sight line of a doctor needs to be switched back and forth between display equipment and a patient, the deformation of human body soft tissues in the operation is difficult to match with a preoperative virtual three-dimensional model and the like. With the development of immersive stereoscopic technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), efforts are being made to apply these new technologies to clinical practice to reduce the interference on surgical navigation systems. Among these technologies, mixed reality technology, as the most efficient and powerful new technology with real-time interaction capability, will be the inevitable trend of immersive intraoperative navigation.
However, the existing virtual reality has technical bottlenecks, so that the virtual three-dimensional model of the preoperative patient is difficult to be practically applied, and the main factor is that the virtual three-dimensional model of the preoperative patient and the actual conditions in the operation may not correspond to each other, and the conditions are particularly frequently generated on soft tissue deformation. For example, in intracranial surgery, the soft tissue collapse caused by factors such as the intracranial pressure change after craniotomy of a patient can cause the preoperative imaging effect to be inconsistent with the actual imaging effect in the operation. In the conventional intraoperative navigation, the problem needs to be corrected by depending on the experience and the field strain capability of a doctor, but the long-time high-intensity surgery and the complicated correction cause great challenges to the physical and psychological aspects of the doctor.
In the existing virtual three-dimensional model self-adaptive deformation solution, a grid-based organization deformation model is a main solution, and the method has the advantages that the model is simple in structure, deformation quantity can be calculated by constructing a physical function between grids, so that the purpose of model correction is achieved, and classical algorithms comprise a Loop subdivision method, a Butterfly subdivision algorithm and the like. However, the method has obvious defects, which are mainly reflected in that the real-time computation amount is large, the model robustness is poor, and the method is difficult to be practically applied to the operation.
With the rapid development of the deep learning technology, more and more neural network algorithms are developed and applied, and have popular results in the actual applications of pathological image super-resolution, pathological tissue image identification, segmentation, detection and the like. However, in the soft tissue adaptive deformation application, the research and the application in the deep learning direction are still insufficient, and compared with the existing solution, the deep learning has the characteristic of strong robustness; the trained network does not require a large amount of computation. Has strong real-time property. At present, deep learning is carried out in similar work, and excellent results are obtained in the self-adaptive deformation of the multi-angle three-dimensional model of the face, so that the reduction rate and the recognition rate of the face can be greatly improved.
Because the adaptive deformation of the model is the main difficulty of judging whether the model approaches to reality in an immersion type operation, and the deep learning method has great potential in MR application, the three-dimensional model adaptive method based on deep learning, which can adapt to complex medical scenes, has important research significance and wide application prospect.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a soft tissue self-adaptive deformation method and system based on mixed reality and 3DGAN, and solves the problem of soft tissue three-dimensional model self-adaptation.
The soft tissue self-adaptive deformation method based on mixed reality and 3DGAN comprises the following steps:
s1: acquiring a preoperative global three-dimensional model and converting the preoperative global three-dimensional model into point cloud data;
s2: in an operation, acquiring local image depth map data of a patient in the operation in real time, and converting the local image depth map data into point cloud data;
s3: sending the point cloud data obtained in the step S1 and the step S2 into a nested 3DGAN network model to realize that the local deformation is mapped to the global deformation;
s4: converting the obtained point cloud global data after deformation into a three-dimensional model,
in step S3, the nested 3DGAN network model includes a local mapping network and a global detection network, the local mapping network deduces the deformation of the global three-dimensional model according to the input local image depth map point cloud data, and the global detection network corrects the three-dimensional model after global deformation output by the local mapping network in this stage according to the preoperative global three-dimensional model point cloud data and the global three-dimensional model point cloud data in the previous stage.
The present invention further improves, before step S1, the method further includes a step of: judging whether to train the nested 3DGAN network model, if so, executing, if not, executing step S1, wherein the step A of training the nested 3DGAN network model comprises the following steps:
a1: obtaining a training set of soft tissue deformation, wherein the training set comprises a plurality of sets of different body membrane models;
a2: training a generator network and a discriminator network of the local mapping network by using a training set until Nash equilibrium is reached;
a3: training a generator network and a discriminator network of the global detection network until Nash equilibrium is reached;
a4: and finishing the training.
The invention is further improved, the local mapping network comprises a local generator network and a local discriminator network arranged at the rear stage of the local generator network, and the processing process of the local generator network is as follows: and after the local image depth map data after point cloud conversion is processed by N layers of 3D convolution block networks, the local image depth map data is processed by a global pooling layer, and finally the deformation of a global three-dimensional model is output, wherein the front N/2 of the 3D convolution block networks adopts cavity convolution, the rear N/2 adopts micro-step convolution, and the active layer of the 3D convolution block networks adopts a ReLU function as an active function.
The invention is further improved, the input of the local mapping network is a three-dimensional point cloud model generated by a local depth image captured by the mixed reality device in the operation; outputting a preliminary global deformation three-dimensional model deduced in the stage; the loss function of the locally mapped network is:
Localloss＝minGmaxDlocalV(Dlocal，G)
＝Ex～Pdata(x)[logDlocal(x)]+Ez～Pz(z)[1-Dlocal(Glocal(z))]
wherein D islocal(x) The expression local discriminator is used for judging whether the output model is matched with the phantom data model or not, the output is a value from 0 to 1, and the larger the value is, the discriminator considers that the difference between the input and the real image is larger; minGmaxDIndicating the need to enable the output of the generator to trick the arbiter, DlocalArbiter representing a local mapping network, GlocalA generator representing a local mapping network, z representing a currently input local three-dimensional point cloud model, and x representing a phantom three-dimensional model of the training set.
The invention is further improved, and the processing process of the global detection network comprises the following steps:
(1) acquiring global three-dimensional model point cloud data output in the previous stage and preoperative global three-dimensional model point cloud data, and combining deformation of a global three-dimensional model preliminarily output by a local mapping network in the current stage;
(2) down-sampling the data of the step (1) through a down-sampling network;
(3) performing feature extraction through a 3D residual convolution network comprising a plurality of residual network layers, performing primary fusion output, and using a LeakyReLU function as an activation function in a pooling activation layer of the 3D residual convolution network;
(4) outputting a corrected point cloud three-dimensional model after up-sampling by an up-sampling network;
(5) and partitioning the point cloud three-dimensional model, and performing feature fusion output.
The invention is further improved, and the processing method for feature fusion in the step (5) comprises the following steps:
(501) dividing the preliminarily fused model into M blocks, and dividing the global three-dimensional model obtained before the operation by the same block size;
(502) respectively passing the N divided preliminary fusion models through a global discriminator of a global detection network;
(503) the global discriminator outputs results corresponding to each block, if the output result of the global discriminator is higher than a set threshold value, the result is directly output, and if the output result of the global discriminator is smaller than the set threshold value, the block data is sent to a block comparison network and a block fusion network of block characteristics;
(504) searching blocks of a global three-dimensional model obtained before an operation by using a block comparison network and a block fusion network, obtaining similarity through inner product, selecting m blocks with the highest similarity value, performing feature fusion of the blocks according to the similarity weight, and finally outputting the adjusted blocks;
(501) and all the blocks are processed by a discriminator to obtain the global deformation three-dimensional point cloud model at the stage.
The invention is further improved, in step (504), the input of the block comparison network is the point cloud block whose global discriminator output result is less than the set threshold value and the preoperative global point cloud block, the block comparison network includes: a shallow convolutional network and a similarity calculation network; the shallow convolutional network comprises 7 convolutional layers, 2 pooling layers and a normalization layer; the calculation formula of the similarity calculation network is as follows:
hi＝<pi/|pi|，qj/|qj|>
wherein p and q represent the extracted point cloud characteristics, i represents the point cloud blocks needing to be adjusted, j represents each compared global point cloud block, the calculation mode is inner product, and the value of i and j is a positive integer between 1 and M.
The invention is further improved, the input of the block fusion network is a similarity matrix, the characteristics of the first m point cloud blocks are extracted from the similarity matrix according to the similarity value from large to small, the block fusion network structure comprises a plurality of layers of lower samples and upper samples, the lower samples adopt cavity convolution, the upper samples adopt sub-pixel convolution, the loss function uses a perception loss function, and the output is the adjusted point cloud blocks.
The invention also provides a system for realizing the soft tissue adaptive deformation method based on mixed reality and 3DGAN, which is characterized by comprising the following steps:
the data acquisition conversion module: the system comprises a preoperative global three-dimensional model, a preoperative patient local image depth map data acquisition unit, a preoperative global three-dimensional model acquisition unit, a preoperative patient local image depth map data acquisition unit and a preoperative patient local image depth map data acquisition unit, wherein the preoperative global three-dimensional model and the intraoperative patient local image depth map data acquisition unit are used for acquiring preoperative global three-dimensional model and intraoperative patient local image depth map data and converting the preoperative global three-dimensional model and the intraoperative patient local image depth map data into point cloud data;
the self-adaptive deformation module: the system comprises a 3DGAN network model, a local deformation mapping module, a point cloud data acquisition module, a point cloud data storage module and a data processing module, wherein the 3DGAN network model is used for acquiring point cloud data;
the second data conversion module: used for converting the obtained point cloud global data after deformation into a three-dimensional model,
the self-adaptive deformation module adopts a nested 3DGAN network model which comprises a local mapping network and a global detection network, wherein the local mapping network deduces the deformation of a global three-dimensional model according to the point cloud data of a local image depth map, and the global detection network corrects the three-dimensional model after global deformation output by the local mapping network at the stage according to the point cloud data of the global three-dimensional model before operation and the point cloud data of the global three-dimensional model at the previous stage.
The invention is further improved, and the invention also comprises a judging module: the method is used for judging whether to train the nested 3DGAN network model or not;
a model training module: for training a nested 3DGAN network model, the model training module comprising:
an acquisition unit: the method comprises the steps of obtaining a training set for soft tissue deformation, wherein the training set comprises a plurality of sets of different body membrane models;
local mapping network training unit: a generator network and a discriminator network for training the local mapping network by using a training set until Nash equilibrium is reached;
the global detection network training unit: the generator network and the discriminator network are used for training the global detection network until Nash equilibrium is reached.
Compared with the prior art, the invention has the beneficial effects that: the method can adapt to complex medical scenes, can enable human body soft tissue deformation in the operation to be matched with a virtual three-dimensional model before the operation according to local images of a patient in the operation process, and can generate the three-dimensional model after the soft tissue in the patient body deforms in real time, so that the condition of a doctor on the anatomical position of the patient in the operation is clear, the safety and success rate of the operation are improved, and the operation burden of the doctor is effectively reduced.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a schematic diagram of a nested 3DGAN network model structure according to the invention;
FIG. 3 is a schematic diagram of a generator portion of a partial mapping network;
fig. 4 is a schematic diagram of a global detection network structure.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
As shown in fig. 1, the adaptive deformation method of the present invention includes the following steps:
starting, judging whether to train a nested 3DGAN (pairwise impedance network) network model, if so, executing a self-adaptive deformation step, and if not, executing a step A of training the nested 3DGAN network model, wherein the method comprises the following steps:
a1: obtaining a training set of soft tissue deformation, wherein the training set comprises a plurality of sets of different body membrane models;
a2: training a generator network and a discriminator network of the local mapping network by using a training set until Nash equilibrium is reached;
a3: training a generator network and a discriminator network of the global detection network until Nash equilibrium is reached;
a4: and finishing the training.
The self-adaptive deformation step comprises the following steps:
s1: acquiring a preoperative global three-dimensional model and converting the preoperative global three-dimensional model into point cloud data;
s2: in an operation, acquiring local image depth map data of a patient in the operation in real time, and converting the local image depth map data into point cloud data;
s3: sending the point cloud data obtained in the step S1 and the step S2 into a nested 3DGAN network model to realize that the local deformation is mapped to the global deformation;
s4: and converting the obtained point cloud global data after deformation into a three-dimensional model.
In this example, the preoperative three-dimensional model of the patient is obtained by three-dimensional reconstruction of two-dimensional image information such as CT or MR before operation, and the intraoperative image of the patient is obtained from a head-mounted mixed reality device of a doctor, such as HoloLens of microsoft. The preoperative global three-dimensional model and intraoperative patient local image depth map data are converted into point cloud formats through CloudCompare free open source software respectively, then the point cloud data are sent to a designed nested 3DGAN method to achieve local deformation mapping to global deformation, and finally the obtained deformed point cloud global data are converted into a three-dimensional model through CloudCompare. Besides the point cloud format, the three-dimensional model file format and the depth map format need to be matched with corresponding head-mounted mixed reality equipment. In the network training process, this example requires a plurality of different sets of body membrane models as the training set for soft tissue deformation.
As shown in fig. 2, the nested 3DGAN network model of this embodiment is based on dcgan (deep adaptive generated adaptive networks), but its convolution kernel is changed to a three-dimensional convolution kernel to adapt to the point cloud format of input and output.
The nested 3DGAN network model comprises a local mapping network and a global detection network, wherein the local mapping network has the function of deducing the deformation of a global three-dimensional model according to an input local point cloud model, and the global detection network has the function of correcting a three-dimensional model after global deformation output by the local mapping network in the stage according to a preoperative global three-dimensional model and a global three-dimensional model in the previous stage, so that the continuity and the authenticity of the model change are ensured.
In the method, the discriminators of two nested networks adopt a patchGAN method, but the difference is that a global detection network adopts a LeakyReLU function as an activation function, and a local mapping network adopts a ReLU function as an activation function. The nesting in the method is embodied in that the global detection network calls a generator of the local detection network to correct the deformation of the global three-dimensional model output by the generator of the local detection network and output the globally deformed three-dimensional model, so that real-time deformed data is generated in the process of acquiring local image data in real time by the head-mounted mixed reality equipment, the soft tissue deformation of a human body in the operation is matched with the preoperative virtual three-dimensional model, and the three-dimensional model after the soft tissue deformation in the patient is generated in real time, so that the condition of a doctor on the anatomical position of the patient in the operation is clear, the safety and success rate of the operation are improved, and the operation burden of the doctor is effectively reduced.
As shown in fig. 3, the processing procedure of the local generator network is as follows: and (3) processing the local image depth map data after point cloud conversion by using an N-layer 3D convolution block network, then processing by using a global pooling layer, and finally outputting the deformation of the global three-dimensional model, wherein the front N/2 of the 3D convolution block network adopts cavity convolution, and the rear N/2 adopts micro-step convolution.
Specifically, the input of the local mapping network of this example is: the mixed reality device is a three-dimensional point cloud model generated by a local depth image captured in an operation. And (3) outputting: and (4) deducing a preliminary global deformation three-dimensional model at the stage.
The loss function is:
Localloss＝minGmaxDlocalV(Dlocal，G)
＝Ex～Pdata(x)[logDlocal(x)]+Ez～Pz(z)[1-Dlocal(Glocal(z))]
wherein D islocal(x) The local discriminator is used for judging whether the output model is consistent with the phantom data model, in the method, the output of the discriminator for generating the countermeasure network is a value from 0 to 1, and the bigger the value is, the discriminator considers that the difference between the input and the real image is bigger; minGmaxDThe expression needs to make the output result of the generator deceive the discriminator, even if its output is minimized, and the discriminator needs to learn the true data distribution to distinguish the true and generated false images, even if the output is maximized, the final result of the training is that the discriminator and the generator reach nash equilibrium. DlocalArbiter representing a local mapping network, GlocalA generator representing a local mapping network, z representing a currently input local three-dimensional point cloud model, and x representing a phantom three-dimensional model of a training setType, in the course of training, DlocalWill participate in the algorithm and update the parameters, after training is completed, only need to calculate GlocalAnd (4) partial.
As shown in fig. 4, the three-dimensional model with high-precision texture generated by the GAN method has a better effect visually, but since the generator network tends to deceive the discriminator network, the high-frequency texture of the generated model is biased to the model in the training data set, so that the detail texture of the generated model and the target point cloud model has a larger deviation in actual use; the method designs a brand-new global monitoring network, and corrects the output of the local mapping network by using modes of feature discrimination, feature fusion and the like to ensure the accuracy of an output soft tissue point cloud model.
The global detection network input of this example: and outputting the global three-dimensional model obtained before the operation and the global network of the previous stage, wherein the stage is the primary global deformation three-dimensional model from the local mapping network.
And (3) outputting: and the global deformation three-dimensional model deduced in the stage.
The loss function is:
Globalloss＝minGmaxDglobalV(Dglobal，G)
＝Ex～Pdata(x)[logDglobal(x)]+Ez～Pz(z)[1-Dglobal(Glocal(z)+xmark*Mark+xglobal*Global)]
wherein global represents the introduction of a global model for preoperative medical image data generation, xglobalThe representation is the weight of the preoperative global model, mark represents the global model finally output in the last stage, xmarkThe representation is the weight of the global model that was finally output from the last stage. The preoperative global model is introduced to ensure the consistency and authenticity of the preoperative and intraoperative models, and the global model output at the last stage is introduced to ensure the continuity of deformation. The method needs to find a balance point of deformation and consistency through multiple times of two weight value adjustments, and generally, x isglobalThe value of (A) needs to be increased continuously with the number of stages。DglobalArbiter representing a global detection network, Glocal(z) represents the preliminary local three-dimensional point cloud model of the current input, and x represents the phantom three-dimensional model of the training set. The generator part of the global detection network adopts residual error network design for preventing overfitting because of more input data.
The intermediate process of this example is:
and (3) outputting the initial global deformation three-dimensional model of the local mapping network at the stage and the global network at the previous stage through the designed 3D residual convolution network for feature fusion output, so as to ensure the output consistency between each stage.
Specifically, the processing procedure of the global detection network is as follows:
(1) acquiring global three-dimensional model point cloud data output in the previous stage and preoperative global three-dimensional model point cloud data, and combining deformation of a global three-dimensional model preliminarily output by a local mapping network in the current stage;
(2) down-sampling the data of the step (1) through a down-sampling network;
(3) extracting characteristics through a 3D residual error convolution network comprising a plurality of residual error network layers, and performing primary fusion output;
(4) outputting a corrected point cloud three-dimensional model after up-sampling by an up-sampling network;
(5) and partitioning the point cloud three-dimensional model, and performing feature fusion output.
As an embodiment of the present invention, the processing method of feature fusion in this embodiment is:
dividing the preliminarily fused model into M blocks, dividing each block into different sizes according to the required model output precision requirement (the sizes of each block are smaller when the precision requirement is higher and the detailed texture is smaller, but the precision requirement is not lower than 7 x 7, because the small point cloud blocks lose the globality, namely the relation with surrounding point clouds), and dividing the global three-dimensional model obtained before the operation by the same block size; respectively enabling M divided preliminary fusion models to pass through a discriminator of the global detection network, enabling the discriminator to output a predicted value, if the predicted value is higher than a set threshold value, considering the block as a final output block, and if the predicted value is lower than the threshold value, sending the block into a block feature comparison and fusion module of the global detection network; the block comparison and fusion network searches blocks of a global three-dimensional model obtained before the operation, obtains similarity through inner product, selects m blocks with the highest similarity value, performs feature fusion of the blocks according to the similarity weight, and finally outputs the adjusted blocks; and all the blocks are processed by a discriminator to obtain the global deformation three-dimensional point cloud model at the stage. And all the blocks are processed by a discriminator to obtain the global deformation three-dimensional point cloud model at the stage.
The penalty function for the discriminator network of this example is:
Globalloss＝minGmaxDV(Dpre，G)
＝Ex～Pdata(x)[logD(x)]+Ez～Pz(z)[1-D(Gpre(z)+xmark*Mark)]
wherein, Gpre(z) represents the output from the local mapping network at this stage, mark represents the global model of the final output at the previous stage, xmarkThe representation is the weight of the global model finally output in the last stage, the output introduced into the global network in the last stage is the continuity of deformation, and x represents the phantom three-dimensional model of the training set.
The output of the global arbiter is a value from 0 to 1, the larger the value is, the higher the confidence of the block is, generally, the threshold is designed to be 0.8, and can be dynamically changed according to the actual requirement.
The input of the block comparison network of this example is a point cloud block with insufficient confidence and a preoperative global point cloud block, the block comparison network includes: a shallow convolutional network and a similarity calculation network; the shallow convolutional network comprises 7 convolutional layers, 2 pooling layers and a normalization layer; the calculation formula of the similarity calculation network is as follows:
hi＝<pi/|pi|，qj/|qj|>
wherein h isiFor the input of the block fusion network, p and q represent the extracted point cloud characteristics, i represents the point cloud blocks needing to be adjusted, j represents each compared global point cloud block,the calculation mode is inner product, i, j takes the value of a positive integer between 1 and M.
The input of the block fusion network is a similarity matrix, the previous m point cloud block features (m is more than or equal to 5, the more smooth texture details are required to be according to the increase of the demand, and the larger m is), are extracted from the similarity matrix from large to small, the block fusion network structure refers to a U-net network (deep learning segmentation network) structure and consists of 8 layers of lower samples and upper samples, the lower samples adopt hole convolution, the upper samples adopt Sub-pixel convolution, and the loss function uses a perception loss function; and outputting the adjusted point cloud blocks.
Through the global detection network, the deviation of the detail texture of the generated model and the target point cloud model in the actual operation is greatly reduced, the generation precision is high, and the requirements of doctors in complex operation scenes are met.
The above-described embodiments are intended to be illustrative, and not restrictive, of the invention, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

Claims (10)

1. The soft tissue self-adaptive deformation method based on mixed reality and 3DGAN is characterized in that: the method comprises the following steps:
s1: acquiring a preoperative global three-dimensional model and converting the preoperative global three-dimensional model into point cloud data;
s2: in an operation, acquiring local image depth map data of a patient in the operation in real time, and converting the local image depth map data into point cloud data;
s3: sending the point cloud data obtained in the step S1 and the step S2 into a nested 3DGAN network model to realize that the local deformation is mapped to the global deformation;
s4: converting the obtained point cloud global data after deformation into a three-dimensional model,
in step S3, the nested 3DGAN network model includes a local mapping network and a global detection network, the local mapping network deduces the deformation of the global three-dimensional model according to the input local image depth map point cloud data, and the global detection network corrects the three-dimensional model after global deformation output by the local mapping network in this stage according to the preoperative global three-dimensional model point cloud data and the global three-dimensional model point cloud data in the previous stage.
2. The soft tissue adaptive deformation method based on mixed reality and 3DGAN of claim 1, wherein: before step S1, the method further includes the step of: judging whether to train the nested 3DGAN network model, if so, executing the step S1, and if not, executing the step A of training the nested 3DGAN network model, wherein the method comprises the following steps:
a1: obtaining a training set of soft tissue deformation, wherein the training set comprises a plurality of sets of different body membrane models;
a2: training a generator network and a discriminator network of the local mapping network by using a training set until Nash equilibrium is reached;
a3: training a generator network and a discriminator network of the global detection network until Nash equilibrium is reached;
a4: and finishing the training.
3. The soft tissue adaptive deformation method based on mixed reality and 3DGAN of claim 1 or 2, wherein: the local mapping network comprises a local generator network and a local discriminator network arranged at the rear stage of the local generator network, and the processing process of the local generator network is as follows: and processing the local image depth map data after point cloud conversion by using an N-layer 3D convolution block network, then performing global pooling layer processing, and finally outputting the deformation of a global three-dimensional model, wherein the front N/2 of the 3D convolution block network adopts cavity convolution, the rear N/2 adopts micro-step convolution, and the active layer of the 3D convolution block network adopts a ReLU function as an active function.
4. The soft tissue adaptive deformation method based on mixed reality and 3DGAN of claim 1 or 2, wherein: the input of the local mapping network is a three-dimensional point cloud model generated by a local depth image captured by the mixed reality equipment in the operation; outputting a preliminary global deformation three-dimensional model deduced in the stage; the loss function of the locally mapped network is:
Localloss＝mincmaxDlocalV(Dlocal,G)
＝Ex～Pdata(x)[IogDlocal(X)]+Ez～Pz(z)[1-Dlocal(Glocal(Z))]
wherein D islocal(x) The expression local discriminator is used for judging whether the output model is matched with the phantom data model or not, the output is a value from 0 to 1, and the larger the value is, the discriminator considers that the difference between the input and the real image is larger; minGmaxDIndicating the need to enable the output of the generator to trick the arbiter, DlocalArbiter representing a local mapping network, GlocalA generator representing a local mapping network, z representing a currently input local three-dimensional point cloud model, and x representing a phantom three-dimensional model of the training set.
5. The soft tissue adaptive deformation method based on mixed reality and 3DGAN of claim 1 or 2, wherein: the processing process of the global detection network comprises the following steps:
(1) acquiring global three-dimensional model point cloud data output in the previous stage and preoperative global three-dimensional model point cloud data, and combining deformation of a global three-dimensional model preliminarily output by a local mapping network in the current stage;
(2) down-sampling the data of the step (1) through a down-sampling network;
(3) performing feature extraction through a 3D residual convolution network comprising a plurality of residual network layers, performing primary fusion output, and using a LeakyReLU function as an activation function in a pooling activation layer of the 3D residual convolution network;
(4) outputting a corrected point cloud three-dimensional model after up-sampling by an up-sampling network;
(5) and partitioning the point cloud three-dimensional model, and performing feature fusion output.
6. The method of claim 5, wherein the soft tissue adaptive deformation based on mixed reality and 3DGAN is characterized in that: the processing method for feature fusion in the step (5) comprises the following steps:
(501) dividing the preliminarily fused model into M blocks, and dividing the global three-dimensional model obtained before the operation by the same block size;
(502) respectively passing the N divided preliminary fusion models through a global discriminator of a global detection network;
(503) the global discriminator outputs results corresponding to each block, if the output result of the global discriminator is higher than a set threshold value, the result is directly output, and if the output result of the global discriminator is smaller than the set threshold value, the block data is sent to a block comparison network and a block fusion network of block characteristics;
(504) searching blocks of a global three-dimensional model obtained before an operation by using a block comparison network and a block fusion network, obtaining similarity through an inner product, selecting m blocks with the highest similarity value, performing feature fusion on the blocks according to the similarity weight, and finally outputting the adjusted blocks;
(505) and all the blocks are processed by a discriminator to obtain the global deformation three-dimensional point cloud model at the stage.
7. The method of claim 6, wherein the soft tissue adaptive deformation based on mixed reality and 3DGAN is characterized in that: in step (504), the input of the block comparison network is a point cloud block and a preoperative global point cloud block, the output result of which is less than a set threshold, and the block comparison network includes: a shallow convolutional network and a similarity calculation network; the shallow convolutional network comprises 7 convolutional layers, 2 pooling layers and a normalization layer; the calculation formula of the similarity calculation network is as follows:
hi＝<pi/|pi|,qj/|qj|>
wherein p and q represent the extracted point cloud characteristics, i represents the point cloud blocks needing to be adjusted, j represents each compared global point cloud block, the calculation mode is inner product, and the value of i and j is a positive integer between 1 and M.
8. The method of claim 7, wherein the soft tissue adaptive deformation based on mixed reality and 3DGAN is characterized in that: the input of the block fusion network is a similarity matrix, the features of the first m point cloud blocks are extracted from the similarity matrix from large to small, the block fusion network structure comprises a plurality of layers of lower samples and upper samples, the lower samples adopt cavity convolution, the upper samples adopt sub-pixel convolution, the loss function uses a perception loss function, and the output is the adjusted point cloud blocks.
9. A system for implementing the mixed reality and 3 DGAN-based soft tissue adaptive deformation method of any one of claims 1-8, comprising:
the data acquisition conversion module: the system comprises a three-dimensional model acquisition unit, a three-dimensional image acquisition unit and a three-dimensional image acquisition unit, wherein the three-dimensional model acquisition unit is used for acquiring a global three-dimensional model before operation and patient local image depth map data during operation and converting the global three-dimensional model and the patient local image depth map data into point cloud data;
the self-adaptive deformation module: the system comprises a 3DGAN network model, a local deformation mapping module, a data acquisition module and a data processing module, wherein the 3DGAN network model is used for acquiring point cloud data;
the second data conversion module: used for converting the obtained point cloud global data after deformation into a three-dimensional model,
the self-adaptive deformation module adopts a nested 3DGAN network model which comprises a local mapping network and a global detection network, wherein the local mapping network deduces the deformation of a global three-dimensional model according to the input local image depth map point cloud data, and the global detection network corrects the three-dimensional model after global deformation output by the local mapping network at the stage according to preoperative global three-dimensional model point cloud data and global three-dimensional model point cloud data at the previous stage.
10. The system of claim 9, wherein: still include the judgement module: the method is used for judging whether to train the nested 3DGAN network model or not;
a model training module: for training a nested 3DGAN network model, the model training module comprising:
an acquisition unit: the method comprises the steps of obtaining a training set for soft tissue deformation, wherein the training set comprises a plurality of sets of different body membrane models;
local mapping network training unit: a generator network and a discriminator network for training the local mapping network by using a training set until Nash equilibrium is reached;
the global detection network training unit: a generator network and a discriminator network for training a global detection network until nash equilibrium is reached.
CN202111372468.XA 2021-11-18 2021-11-18 Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN Pending CN114092643A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111372468.XA CN114092643A (en) 2021-11-18 2021-11-18 Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111372468.XA CN114092643A (en) 2021-11-18 2021-11-18 Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN

Publications (1)

Publication Number Publication Date
CN114092643A true CN114092643A (en) 2022-02-25

Family

ID=80302210

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111372468.XA Pending CN114092643A (en) 2021-11-18 2021-11-18 Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN

Country Status (1)

Country Link
CN (1) CN114092643A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116433812A (en) * 2023-06-08 2023-07-14 海马云（天津）信息技术有限公司 Method and device for generating virtual character by using 2D face picture

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116433812A (en) * 2023-06-08 2023-07-14 海马云（天津）信息技术有限公司 Method and device for generating virtual character by using 2D face picture
CN116433812B (en) * 2023-06-08 2023-08-25 海马云（天津）信息技术有限公司 Method and device for generating virtual character by using 2D face picture

Similar Documents

Publication Publication Date Title
CN109285215B (en) Human body three-dimensional model reconstruction method and device and storage medium
CN112200843B (en) Super-voxel-based CBCT and laser scanning point cloud data tooth registration method
KR20210051141A (en) Method, apparatus and computer program for providing augmented reality based medical information of patient
CN112614169B (en) 2D/3D spine CT (computed tomography) level registration method based on deep learning network
JP7190059B2 (en) Image matching method, apparatus, device and storage medium
CN108618749B (en) Retina blood vessel three-dimensional reconstruction method based on portable digital fundus camera
CN111091616B (en) Reconstruction method and device of three-dimensional ultrasonic image
CN111080778B (en) Online three-dimensional reconstruction method of binocular endoscope soft tissue image
KR20210052270A (en) Method, apparatus and computer program for providing augmented reality based medical information of patient
CN114092643A (en) Soft tissue self-adaptive deformation method based on mixed reality and 3DGAN
CN114399527A (en) Method and device for unsupervised depth and motion estimation of monocular endoscope
CN110223396B (en) Morphology-based spine simulation correction method and device
CN115619790B (en) Hybrid perspective method, system and equipment based on binocular positioning
CN116958443A (en) SMPLX-based digital human quantitative detection model reconstruction method and application
CN116993805A (en) Intraoperative residual organ volume estimation system oriented to operation planning assistance
Chen et al. The research and practice of medical image enhancement and 3D reconstruction system
Zhao et al. Research on 3D reconstruction algorithm of medical CT image based on parallel contour
Zhou et al. A research of surgical navigation system of mixed reality based on image registration
CN114049442A (en) Three-dimensional face sight line calculation method
CN109741360B (en) Bone joint segmentation method, device, terminal and readable medium
JP5403431B2 (en) Tomographic image processing method and apparatus
CN111598904A (en) Image segmentation method, device, equipment and storage medium
WO2024050918A1 (en) Endoscope positioning method, electronic device, and non-transitory computer-readable storage medium
Robb et al. Efficient shape-based algorithms for modeling patient specific anatomy from 3D medical images: applications in virtual endoscopy and surgery
CN116421310A (en) Internal safety distance identification system oriented to minimally invasive anatomical path constraint

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination