CN117557724A - Head presentation method and system for brain surgery patient based on pose estimation - Google Patents
Head presentation method and system for brain surgery patient based on pose estimation Download PDFInfo
- Publication number
- CN117557724A CN117557724A CN202311528057.4A CN202311528057A CN117557724A CN 117557724 A CN117557724 A CN 117557724A CN 202311528057 A CN202311528057 A CN 202311528057A CN 117557724 A CN117557724 A CN 117557724A
- Authority
- CN
- China
- Prior art keywords
- brain
- model
- vein
- patient
- pose
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000004556 brain Anatomy 0.000 title claims abstract description 207
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000001356 surgical procedure Methods 0.000 title claims abstract description 15
- 210000003462 vein Anatomy 0.000 claims abstract description 146
- 210000000988 bone and bone Anatomy 0.000 claims abstract description 47
- 238000012549 training Methods 0.000 claims abstract description 43
- 210000003625 skull Anatomy 0.000 claims abstract description 33
- 238000005520 cutting process Methods 0.000 claims abstract description 22
- 238000007428 craniotomy Methods 0.000 claims abstract description 10
- 210000004298 cerebral vein Anatomy 0.000 claims description 44
- 239000013598 vector Substances 0.000 claims description 18
- 210000005013 brain tissue Anatomy 0.000 claims description 16
- 238000004422 calculation algorithm Methods 0.000 claims description 15
- 238000003860 storage Methods 0.000 claims description 12
- 230000000007 visual effect Effects 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 10
- 210000001367 artery Anatomy 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 8
- 230000035040 seed growth Effects 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000004519 manufacturing process Methods 0.000 claims description 6
- 210000001519 tissue Anatomy 0.000 claims description 6
- FPIGOBKNDYAZTP-UHFFFAOYSA-N 1,2-epoxy-3-(4-nitrophenoxy)propane Chemical compound C1=CC([N+](=O)[O-])=CC=C1OCC1OC1 FPIGOBKNDYAZTP-UHFFFAOYSA-N 0.000 claims description 4
- 210000004204 blood vessel Anatomy 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 210000000056 organ Anatomy 0.000 claims description 4
- 238000009877 rendering Methods 0.000 claims description 4
- 230000003321 amplification Effects 0.000 claims description 3
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 3
- 230000002093 peripheral effect Effects 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 description 7
- 210000003491 skin Anatomy 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 230000004927 fusion Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 210000003484 anatomy Anatomy 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 206010051290 Central nervous system lesion Diseases 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 208000015181 infectious disease Diseases 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000012010 growth Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000007493 shaping process Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013334 tissue model Methods 0.000 description 1
- 230000002792 vascular Effects 0.000 description 1
Abstract
The invention provides a head presentation method and a head presentation system for a brain surgery patient based on pose estimation, wherein the method comprises the following steps: acquiring preoperative head two-dimensional image data of a patient, and reconstructing a brain model of the patient; pre-cutting the brain vein sub-model through the skull sub-model by adopting Boolean operation, and after a target operation path is selected, determining the bone window position and the bone window size of the brain model, finishing the cutting treatment of the brain vein model; creating a brain vein pose estimation network model and a pose estimation training data set, and iteratively training the brain vein pose estimation network model; the RGB image of the craniotomy area of the patient is acquired, input into a brain vein pose estimation network model to acquire the brain vein pose of the patient, the brain vein pose is fused with the head pose of the patient to perform virtual reality presentation, and the real-time update of the virtual reality presentation is controlled through a preset gesture.
Description
Technical Field
The invention relates to the technical field of medical image processing and surgical application, in particular to a head presentation method and a head presentation system for a brain surgery patient based on pose estimation.
Background
The brain surgery has the characteristics of high risk, high difficulty and the like, a surgeon needs to check a medical image of a patient through a computer screen and simulate the focus position in the brain so as to plan a surgery path, however, the brain surgery needs to have professional knowledge, good space simulation capability and rich experience due to the fact that the information presentation mode of non-direct vision is observed through the computer screen, and the brain surgery has high difficulty.
In order to solve the problem, an augmented reality technology is introduced in the prior art, 3D anatomy structure and guide information of a patient are superimposed on the head of the patient in real time, a doctor can effectively improve efficiency and precision of operation by directly viewing the head of the patient and superimposed augmented reality presentation information, wherein real-time and accurate registration of a 3D head model and a real patient head is taken as an important technical support, and is mainly realized by adopting artificial marks and based on facial features at present, however, the real-time and accurate registration technology of the 3D head model and the real patient head has the following defects:
(1) The manual marking method needs to be prefabricated and fixed on the head of the patient, which causes inconvenience to the patient and is unfavorable for the shaping of the sterile environment.
(2) Due to the need for a sterile environment, the face of the intraoperative patient is obscured and real-time intraoperative registration cannot be accomplished using facial features.
Therefore, there is a need for a head presentation method for a brain surgery patient based on pose estimation to solve the problems of low accuracy of registration between the 3D brain structure of the patient and the real head, poor real-time performance, and low virtual reality fusion presentation performance.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a head presentation method and a head presentation system for a brain surgery patient based on pose estimation.
The first aspect of the invention discloses a head presentation method of a brain surgery patient based on pose estimation, which comprises the following steps:
s1, acquiring preoperative head two-dimensional image data of a patient, and reconstructing the two-dimensional image data into a brain model of the patient based on three-dimensional reconstruction, wherein the brain model comprises a skin submodel, a skull submodel, a brain tissue submodel, a brain artery submodel, a brain vein submodel and a focus submodel;
s2: pre-cutting the brain vein sub-model through the skull sub-model by adopting Boolean operation, determining the bone window position and the bone window size of the brain model after selecting a target operation path, and finishing cutting of the brain vein model according to the bone window position and the bone window size;
s3: creating a brain vein pose estimation network model and a pose estimation training data set, and iteratively training the brain vein pose estimation network model through the pose estimation training data set;
S4: and acquiring RGB images of an intraoperative patient craniotomy area, inputting the RGB images into the brain vein pose estimation network model to acquire the brain vein pose of the patient, fusing the brain vein pose with the head pose of the patient to perform virtual reality presentation, and controlling real-time update of the virtual reality presentation through preset gestures.
In an alternative embodiment, the pre-clipping the brain vein sub-model by the skull sub-model using boolean operations includes:
s201: after confirming the central point coordinates of the skull sub-model, scaling the skull sub-model in an equal ratio;
s202: the skull sub-model is configured as a reference object in a Boolean operation plug-in, the brain vein sub-model is configured as a subtracted object, the subtracted object is pre-cut through the reference object, and the pre-cutting comprises cutting the brain vein sub-model on the inner side of brain tissue of a patient.
In an alternative embodiment, the determining the location and size of the bone window of the brain model after the selecting the target surgical path includes:
s203: and determining the geometric center of the focus submodel, setting the geometric center as a starting point Vg, and establishing a search space according to the head of the patient.
S204: setting tissues and organs of the head of a patient as obstacles, defining a safe distance between the target operation path and the obstacles, and searching the target operation path and a target path endpoint Vt in the search space by adopting an intelligent optimization algorithm;
s205: connecting the start point Vg and the target path end point Vt to obtain a path vectorAnd orthogonally projecting the focus submodel to the brain tissue surface of the patient along the path vector direction, and determining the bone window position and the bone window size after configuring the projection edge as a mask.
In an alternative embodiment, the clipping of the brain vein model according to the bone window position and the bone window size includes:
s206: by setting the edge of the mask after the equal ratio amplification as a clipping boundary, and then vector along the pathIs cut for the cerebral vein sub-model.
In an alternative embodiment, the creating the pose estimation training dataset comprises:
s301: selecting a blood vessel peripheral point as a seed growth starting point according to the point set of the cut brain vein sub-model to construct a brain vein model, and presetting a brain vein bifurcation judging principle for judging whether bifurcation occurs or not based on the seed growth starting point;
S302: extracting an isoplane value of a brain vein blood vessel model through a marching cube algorithm to obtain a brain vein blood vessel surface network;
s303: judging the brain vein surface network according to the brain vein bifurcation judging principle, confirming bifurcation points of all branches of the brain vein, marking the bifurcation points and selecting the bifurcation points as key points;
s304: traversing each branch of the cerebral vein model until all branches corresponding to the bifurcation points are cut off, and storing the key points and the central points of the cerebral vein sub-model to a three-dimensional point set K 3d In (a) and (b);
s305: introducing a cerebral vein sub-model of the cerebral vein model containing the resected branches into a three-dimensional graphics processing engine to manufacture a data set;
s306: parallel projection of the cerebral vein sub-model and collection of the three-dimensional points K 3d Projecting into a two-dimensional RGB image to obtain the brain vein sub-model as a two-dimensional RGB image and a set K 3d Two-dimensional point set K of (2) 2d ；
S307: constructing a mask area of a two-dimensional RGB image and a three-dimensional point set, obtaining RGB images of brain vein sub-models under different visual angles by changing camera view point rendering, and recording pose matrixes P of the corresponding brain vein sub-models under all camera view points render ；
S308, adding a brain vein semantic excision tag, a brain vein two-dimensional point set and a pose and pose matrix P to the two-dimensional RGB image in the data set render A pose estimation training dataset is obtained.
In an alternative embodiment, the iteratively training the brain vein pose estimation network model by the pose estimation training dataset comprises:
s309: inputting the pose estimation training data set into a brain vein pose estimation network model, and outputting a class label of each pixel in the RGB image through a backbone network, wherein the class label comprises a unit vector field for judging whether the pixel belongs to a brain vein and expressing the direction of the pixel towards a key point;
s310: generating a set of assumed key points for all the key points, wherein the assumed key points are two pixels selected randomly, and the intersection point of pixel vectors is used as the assumed key point;
s311: observing each set of hypothesized key points in sequence by meeting the condition of highest confidence, and selecting the highest scoring point in each set of hypothesized key points as a target key point of the brain vein pose estimation network model;
s312: and solving the pose P of the brain vein in the RGB image input into the brain vein pose estimation network model through an EPNP algorithm based on a preset three-dimensional key point and a target key point.
In an optional embodiment, the calculation formula for obtaining the brain vein pose of the patient by inputting the RGB image into the brain vein pose estimation network model is:
after the images are sequentially input into the trained network model, each image can obtain a pixel and pose matrix P belonging to the cerebral veins, and each image is segmented according to the time sequence of the input images and the semantics of the imagesUpdating the actual cerebral vein pose P of a patient once word Wherein N is pixel-1 、N pixel -2、......、N pixel-n The number of pixels belonging to the cerebral vein in the nth image inputted within 0 to 0.5s is represented.
In a second aspect, the invention discloses a brain surgical patient head presentation system based on pose estimation, the system comprising:
the preoperative model reconstruction module is used for acquiring preoperative head two-dimensional image data of a patient, reconstructing the two-dimensional image data into a brain model of the patient based on three-dimensional reconstruction, wherein the brain model comprises a skin submodel, a skull submodel, a brain tissue submodel, a brain artery submodel, a brain vein submodel and a focus submodel;
the preoperative model processing module is used for pre-cutting the cerebral vein sub-model through the skull sub-model by adopting Boolean operation, determining the bone window position and the bone window size of the brain model after selecting a target operation path, and finishing the cutting processing of the cerebral vein model according to the bone window position and the bone window size;
The preoperative model training module is used for creating a brain vein pose estimation network model and a pose estimation training data set, and iteratively training the brain vein pose estimation network model through the pose estimation training data set;
the head presenting model of the patient in operation is used for acquiring RGB images of the craniotomy area of the patient in operation, inputting the RGB images into the brain vein pose estimating network model to acquire the brain vein pose of the patient, fusing the brain vein pose with the head pose of the patient to carry out virtual reality presentation, and controlling real-time update of the virtual reality presentation through preset gestures.
A third aspect of the present invention discloses a brain surgery patient head presenting device based on pose estimation, comprising:
at least one processor, and,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the pose estimation based brain surgical patient head presentation method according to any of the first aspect of the present invention.
A fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method for presenting a head of a brain surgical patient based on pose estimation according to any of the first aspect of the present invention.
Compared with the prior art, the invention has the following advantages:
according to the invention, under a large-area covering scene of the head of a brain operation patient, the pose estimation is carried out on the head of the patient by only adopting the natural brain vein features extracted from the RGB camera with the AR head display and the constructed neural network model, and the model is cut through the priori knowledge of the focus position, so that the pose estimation accuracy and efficiency of the neural network model are improved, the fusion effect of the head of the patient and the brain model in the operation is realized without depending on the current mainstream method of using the artificial mark and the depth camera, the operation infection risk is reduced, the cost is reduced, and simultaneously, a three-dimensional visible brain lesion structure and important tissues are presented for doctors, thereby improving the operation precision and efficiency.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
FIG. 1 is a flow chart of a method of presenting a head of a brain surgical patient based on pose estimation according to the present invention;
fig. 2 is a schematic diagram of a brain surgical patient head presentation system based on pose estimation of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described and illustrated below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden on the person of ordinary skill in the art based on the embodiments provided herein, are intended to be within the scope of the present application.
It is apparent that the drawings in the following description are only some examples or embodiments of the present application, and it is possible for those of ordinary skill in the art to apply the present application to other similar situations according to these drawings without inventive effort. Moreover, it should be appreciated that while such a development effort might be complex and lengthy, it would nevertheless be a routine undertaking of design, fabrication, or manufacture for those of ordinary skill having the benefit of this disclosure, and thus should not be construed as having the benefit of this disclosure.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly and implicitly understood by those of ordinary skill in the art that the embodiments described herein can be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar terms herein do not denote a limitation of quantity, but rather denote the singular or plural. The terms "comprising," "including," "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to only those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "connected," "coupled," and the like in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as used herein refers to two or more. "and/or" describes an association relationship of an association object, meaning that there may be three relationships, e.g., "a and/or B" may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. The terms "first," "second," "third," and the like, as used herein, are merely distinguishing between similar objects and not representing a particular ordering of objects.
Example 1
Referring to fig. 1, an embodiment of the invention discloses a head presenting method of a brain surgery patient based on pose estimation, comprising the following steps:
s1, acquiring preoperative head two-dimensional image data of a patient, and reconstructing the two-dimensional image data into a brain model of the patient based on three-dimensional reconstruction, wherein the brain model comprises a skin submodel, a skull submodel, a brain tissue submodel, a brain artery submodel, a brain vein submodel and a focus submodel;
it should be noted that, two-dimensional image data of the head (including the skin, skull, brain tissue, brain vein, brain artery and focus) of the patient is acquired through CT and MR devices before operation, and three-dimensional reconstruction is performed on the head of the patient by using three-dimensional medical imaging software to obtain a three-dimensional model of the head of the patient (including a skin model, a skull model, a brain tissue model, a brain artery and vein model and a focus model).
S2: pre-cutting the brain vein sub-model through the skull sub-model by adopting Boolean operation, determining the bone window position and the bone window size of the brain model after selecting a target operation path, and finishing cutting of the brain vein model according to the bone window position and the bone window size;
It should be noted that, due to the special tree structure of the cerebral veins, when the complete three-dimensional model of the cerebral veins is projected and rendered to a two-dimensional image in the process of making a data set, the branches of different veins are overlapped to form a pseudo-vein branch. Therefore, the brain vein model needs to be cut, and the three-dimensional brain vein model needs to be cut before the training set is manufactured, so that in the implementation, the three-dimensional brain vein model of a corresponding area is cut out by planning the approximate bone window position according to the priori knowledge of the position of a focus relative to the head in the three-dimensional head model, and the calculation efficiency of the neural network model and the accuracy degree of estimating the pose are improved.
In an alternative embodiment, the pre-clipping the brain vein sub-model by the skull sub-model using boolean operations includes:
s201: after confirming the central point coordinates of the skull sub-model, scaling the skull sub-model in an equal ratio;
s202: the skull sub-model is configured as a reference object in a Boolean operation plug-in, the brain vein sub-model is configured as a subtracted object, the subtracted object is pre-cut through the reference object, and the pre-cutting comprises cutting the brain vein sub-model on the inner side of brain tissue of a patient.
Furthermore, the skull model and the cerebral veins of the patient obtained after the three-dimensional reconstruction are imported into a local project of Un's strength, and CSG plug-in units are installed in the environment of the local project. The skull model in the head model was replicated and was set to 1:0.7 to 1: scaling down in an equal ratio of 0.6, and simultaneously enabling the coordinates of the central point under the world coordinate system of the original skull model and the world coordinate system of the scaled-down skull model to be the same. In CSG, a reduced skull model is set as Brush, a brain vein model is set as Target, and the brain vein model is cut out with the reduced skull model by boolean operation. The brain vein model located inside the brain tissue was cut as much as possible.
In an alternative embodiment, the determining the location and size of the bone window of the brain model after the selecting the target surgical path includes:
s203: and determining the geometric center of the focus submodel, setting the geometric center as a starting point Vg, and establishing a search space according to the head of the patient.
S204: setting tissues and organs of the head of a patient as obstacles, defining a safe distance between the target operation path and the obstacles, and searching the target operation path and a target path endpoint Vt in the search space by adopting an intelligent optimization algorithm;
S205: connecting the start point Vg and the target path end point Vt to obtain a path vectorAnd orthogonally projecting the focus submodel to the brain tissue surface of the patient along the path vector direction, and determining the bone window position and the bone window size after configuring the projection edge as a mask.
Preferably, an intelligent optimization algorithm (such as a particle algorithm and an ant colony algorithm) is utilized, a geometric center Vg of the three-dimensional focus model is taken as a starting point, a search space is established according to the head size of a patient, and an optimal operation path for planning to reach the skin surface of the head of the patient is found. The critical tissues and organs within the head are set as obstacles and define the safe distance of the path (i.e., the distance between the path and the obstacle boundary) to find the optimal path endpoint Vt. Edge of the frameThe direction, the focus model is projected to the brain tissue surface through orthographic projection, the projection edge is used as a mask, the mask is used for planning the position and the size of the bone window, and the person skilled in the art can easily know that the method can also plan the position and the size of the bone window according to the guidance of doctors.
In an alternative embodiment, the clipping of the brain vein model according to the bone window position and the bone window size includes:
S206: by setting the edge of the mask after the equal ratio amplification as a clipping boundary, and then vector along the pathIs cut for the cerebral vein sub-model.
The edge may be defined by an edge in which the mask is enlarged by 1.5 times in equal ratioAnd cutting the three-dimensional cerebral vein model in the direction.
S3: creating a brain vein pose estimation network model and a pose estimation training data set, and iteratively training the brain vein pose estimation network model through the pose estimation training data set;
in an alternative embodiment, the creating the pose estimation training dataset comprises:
s301: selecting a blood vessel peripheral point as a seed growth starting point according to the point set of the cut brain vein sub-model to construct a brain vein model, and presetting a brain vein bifurcation judging principle for judging whether bifurcation occurs or not based on the seed growth starting point;
s302: extracting an isoplane value of a brain vein blood vessel model through a marching cube algorithm to obtain a brain vein blood vessel surface network;
s303: judging the brain vein surface network according to the brain vein bifurcation judging principle, confirming bifurcation points of all branches of the brain vein, marking the bifurcation points and selecting the bifurcation points as key points;
S304: traversing each branch of the cerebral vein model until all branches corresponding to the bifurcation points are cut off, and storing the key points and the central points of the cerebral vein sub-model to a three-dimensional point set K 3d In (a) and (b);
s305: introducing a cerebral vein sub-model of the cerebral vein model containing the resected branches into a three-dimensional graphics processing engine to manufacture a data set;
s306: parallel projection of the cerebral vein sub-model and collection of the three-dimensional points K 3d Projecting into a two-dimensional RGB image to obtain the brain vein sub-model as a two-dimensional RGB image and a set K 3d Two-dimensional point set K of (2) 2d ；
S307: constructing a mask area of a two-dimensional RGB image and a three-dimensional point set, obtaining RGB images of brain vein sub-models under different visual angles by changing camera view point rendering, and recording pose matrixes P of the corresponding brain vein sub-models under all camera view points render ；
S308, adding a brain vein semantic excision tag, a brain vein two-dimensional point set and a pose and pose matrix P to the two-dimensional RGB image in the data set render A pose estimation training dataset is obtained.
In the process of cuttingSelecting a vascular tip point or a cutting edge point from the cut cerebral vein model point set as a seed growth starting point, growing in a direction away from the starting point, stopping growing when encountering bifurcation, wherein for cerebral vein bifurcation judgment: extracting the equivalent surface of the vein blood vessel model from the model by using a MarchingCubes algorithm (marching cube algorithm) to obtain a vein blood vessel surface grid, and judging the occurrence of blood vessel bifurcation by checking the equivalent surface: when all points in the isosurface are 26-neighborhood connected, then no bifurcation of the isosurface occurs; all points of the isosurface cannot be connected in a 26-neighborhood mode, namely the isosurface is not communicated, bifurcation is generated on the isosurface with the distance value, the bifurcation point of the current branch is selected as a key point, the branch is marked, the marked branch vein blood vessel is removed after each growth, other seeds continue to grow towards the branch without marking, and when no branch can grow, all vein blood vessel branches in the traversal model are obtained. Adding the obtained key points and central points of the brain vein model into a set K 3d In (K) 3d The number of (a) may be chosen between 10 and 12.
In an alternative embodiment, the iteratively training the brain vein pose estimation network model by the pose estimation training dataset comprises:
s309: inputting the pose estimation training data set into a brain vein pose estimation network model, and outputting a class label of each pixel in the RGB image through a backbone network, wherein the class label comprises a unit vector field for judging whether the pixel belongs to a brain vein and expressing the direction of the pixel towards a key point;
s310: generating a set of assumed key points for all the key points, wherein the assumed key points are two pixels selected randomly, and the intersection point of pixel vectors is used as the assumed key point;
s311: observing each set of hypothesized key points in sequence by meeting the condition of highest confidence, and selecting the highest scoring point in each set of hypothesized key points as a target key point of the brain vein pose estimation network model;
s312: and solving the pose P of the brain vein in the RGB image input into the brain vein pose estimation network model through an EPNP algorithm based on a preset three-dimensional key point and a target key point.
Further, the model is imported into three-dimensional graphic image software such as a blender engine to manufacture a data set, parallel projection is carried out on the model, the calculated three-dimensional key point set K3d is projected into a two-dimensional RGB image, a two-dimensional RGB image and a two-dimensional key point set K2d are obtained, and meanwhile a corresponding mask area is established. Changing camera viewpoint to render to obtain RGB images of brain veins under different visual angles, and recording camera pose matrix P of brain vein model under different visual angles render 。
The data set comprises 9000 to 10000 640x840 two-dimensional RGB images obtained through rendering, and brain vein semantic segmentation labels, brain vein two-dimensional key point true values and brain vein pose matrix P corresponding to each RGB image render . Meanwhile, in order to enable the robustness of the neural network model to be better, the neural network model can adapt to an actual operation environment, data enhancement is carried out on a data set, a large number of operation environment backgrounds are utilized to replace model background pictures, the brightness of the pictures is randomly adjusted, a mask area is randomly shielded, and the shielding range is randomly set to be 1/5 to 1/3 of the size of the mask area.
It should be noted that, the training set 640x840 two-dimensional RGB image is input into the PVNet network for training, and the class label of each pixel (in this embodiment, the class is divided into two classes only, that is, whether the pixel belongs to the pixels composing the cerebral vein) is output through the backhaul (modified res net 18) and one unit vector field is generated for each key point. Where the unit vector represents the direction in which pixels belonging to the brain vein image tend to be keypoints. After the directions of all pixels belonging to the cerebral veins pointing to different key points in the current input image are obtained, randomly selecting two pixels for each key point, taking the intersection point of the two pixel vectors as an assumed key point, repeating N times, generating a set of assumed points for each key point, voting the two-dimensional key points of each set of assumption, and selecting the prediction point with the highest confidence. And taking the point with the highest score in each group as a two-dimensional key point obtained by final network calculation. And solving the pose P of the cerebral veins in the current input RGB image by using an EPNP algorithm through the pre-selected three-dimensional key points and the corresponding two-dimensional key points obtained by calculation of the network model, wherein 180 epochs are set in total in the specific training process of the network, and the initial learning rate is set to be 0.001. The learning rate was halved after each 15 epochs training.
S4: and acquiring RGB images of an intraoperative patient craniotomy area, inputting the RGB images into the brain vein pose estimation network model to acquire the brain vein pose of the patient, fusing the brain vein pose with the head pose of the patient to perform virtual reality presentation, and controlling real-time update of the virtual reality presentation through preset gestures.
In an optional embodiment, the calculation formula for obtaining the brain vein pose of the patient by inputting the RGB image into the brain vein pose estimation network model is:
after the images are sequentially input into the trained network model, each image can obtain a pixel and pose matrix P belonging to the cerebral veins, and each image is segmented according to the time sequence of the input images and the semantics of the imagesUpdating the actual cerebral vein pose P of a patient once word Wherein N is pixel-1 、N pixel -2、......、N pixel-n The number of pixels belonging to the cerebral vein in the nth image inputted within 0 to 0.5s is represented.
The RGB image of the craniotomy region under the current visual angle is acquired, and the RGB image of the craniotomy region under the current visual angle is acquired by using a visual sensing device. The method uses a camera of a Microsoft AR head-mounted display Hololens2 to collect images, and other AR head-mounted devices can be adopted in specific implementation. In the operation process, when a doctor wears the head display, a color image of the craniotomy area under the current doctor visual angle is acquired through a color camera of the head display. The acquisition frequency f of the camera is set at 6-12 sheets/s and is set to an even number.
Further, AR virtual-real fusion presentation is performed to obtain actual cerebral vein pose P of patient render Representing the actual head pose of the patient, registering the virtual head model of the patient with the head of the patient, and realizing the fusion effect of the virtual and the real. The pose of the virtual head model of the patient is according to P every 0.5s render The method is adjusted once, meanwhile, an interaction method is provided by utilizing the gesture recognition function of the head display, a doctor can observe different anatomical structures included in the model through the gesture interaction function, when the head display loses the visual field of a bone window area in operation or the exposed area of the bone window area and the visual field of a head display camera are too small, the accurate head pose of a patient cannot be obtained, and the virtual model can drift in an actual space. Therefore, the embodiment adopts the anchor point labeling method based on the space SLAM technology, the head-display camera is utilized to automatically scan and map the operation environment at one time, when images acquired for more than 3 seconds continuously all have the anatomical structure of cerebral veins, the registration of the virtual head model of the patient and the actual pose of the patient is judged, the anchor point is automatically added at the head position of the actual patient, after the anchor point is generated, the head-display stops transmitting RGB images to the neural network for pose estimation, and the pose of the head model keeps the current state.
Furthermore, after the doctor adjusts the head of the patient during updating the pose of the patient, the anchor point can be canceled through the preset gesture. The head display can continuously shoot RGB images and transmit the RGB images to the PC end to re-estimate the position and the pose of the head of the patient, so that the position and the pose of the model in the real space are updated.
According to the invention, under a large-area covering scene of the head of a brain operation patient, the pose estimation is carried out on the head of the patient by only adopting the natural brain vein features extracted from the RGB camera with the AR head display and the constructed neural network model, and the model is cut through the priori knowledge of the focus position, so that the pose estimation accuracy and efficiency of the neural network model are improved, the fusion effect of the head of the patient and the brain model in the operation is realized without depending on the current mainstream method of using the artificial mark and the depth camera, the operation infection risk is reduced, the cost is reduced, and simultaneously, a three-dimensional visible brain lesion structure and important tissues are presented for doctors, thereby improving the operation precision and efficiency.
As shown in fig. 2, a second aspect of the present invention discloses a brain surgical patient head presentation system based on pose estimation, the system comprising:
the preoperative model reconstruction module is used for acquiring preoperative head two-dimensional image data of a patient, reconstructing the two-dimensional image data into a brain model of the patient based on three-dimensional reconstruction, wherein the brain model comprises a skin submodel, a skull submodel, a brain tissue submodel, a brain artery submodel, a brain vein submodel and a focus submodel;
The preoperative model processing module is used for pre-cutting the cerebral vein sub-model through the skull sub-model by adopting Boolean operation, determining the bone window position and the bone window size of the brain model after selecting a target operation path, and finishing the cutting processing of the cerebral vein model according to the bone window position and the bone window size;
the preoperative model training module is used for creating a brain vein pose estimation network model and a pose estimation training data set, and iteratively training the brain vein pose estimation network model through the pose estimation training data set;
the head presenting model of the patient in operation is used for acquiring RGB images of the craniotomy area of the patient in operation, inputting the RGB images into the brain vein pose estimating network model to acquire the brain vein pose of the patient, fusing the brain vein pose with the head pose of the patient to carry out virtual reality presentation, and controlling real-time update of the virtual reality presentation through preset gestures.
A third aspect of the present invention discloses a brain surgery patient head presenting device based on pose estimation, comprising:
at least one processor, and,
a memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the pose estimation based brain surgical patient head presentation method according to any of the first aspect of the present invention.
The computer device may be a terminal comprising a processor, a memory, a network interface, a display screen and input means connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method of brain surgical patient head presentation based on pose estimation. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
A fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method for presenting a head of a brain surgical patient based on pose estimation according to any of the first aspect of the present invention.
Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by way of a computer program, which may be stored on a non-volatile computer readable storage medium, which when executed may include the above-described embodiments of a method for presenting a head of a brain surgical patient via pose-based estimation. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
Alternatively, the above modules of the present invention may be stored in a computer-readable storage medium if implemented as software functional modules and sold or used as a separate product. Based on such understanding, the technical solution of the embodiments of the present invention may be essentially or part contributing to the related art, and the computer software product may be stored in a storage medium, and include several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the methods of the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program code, such as a removable storage device, RAM, ROM, magnetic or optical disk.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (10)
1. A method of presenting a head of a brain surgical patient based on pose estimation, the method comprising:
s1, acquiring preoperative head two-dimensional image data of a patient, and reconstructing the two-dimensional image data into a brain model of the patient based on three-dimensional reconstruction, wherein the brain model comprises a skin submodel, a skull submodel, a brain tissue submodel, a brain artery submodel, a brain vein submodel and a focus submodel;
S2: pre-cutting the brain vein sub-model through the skull sub-model by adopting Boolean operation, determining the bone window position and the bone window size of the brain model after selecting a target operation path, and finishing cutting of the brain vein model according to the bone window position and the bone window size;
s3: creating a brain vein pose estimation network model and a pose estimation training data set, and iteratively training the brain vein pose estimation network model through the pose estimation training data set;
s4: and acquiring RGB images of an intraoperative patient craniotomy area, inputting the RGB images into the brain vein pose estimation network model to acquire the brain vein pose of the patient, fusing the brain vein pose with the head pose of the patient to perform virtual reality presentation, and controlling real-time update of the virtual reality presentation through preset gestures.
2. The method of claim 1, wherein pre-clipping the brain vein sub-model by the skull sub-model using boolean operations comprises:
s201: after confirming the central point coordinates of the skull sub-model, scaling the skull sub-model in an equal ratio;
S202: the skull sub-model is configured as a reference object in a Boolean operation plug-in, the brain vein sub-model is configured as a subtracted object, the subtracted object is pre-cut through the reference object, and the pre-cutting comprises cutting the brain vein sub-model on the inner side of brain tissue of a patient.
3. The method of claim 2, wherein determining the bone window position and bone window size of the brain model after selecting the target surgical path comprises:
s203: and determining the geometric center of the focus submodel, setting the geometric center as a starting point Vg, and establishing a search space according to the head of the patient.
S204: setting tissues and organs of the head of a patient as obstacles, defining a safe distance between the target operation path and the obstacles, and searching the target operation path and a target path endpoint Vt in the search space by adopting an intelligent optimization algorithm;
s205: connecting the start point Vg and the target path end point Vt to obtain a path vectorAnd orthogonally projecting the focus submodel to the brain tissue surface of the patient along the path vector direction, and determining the bone window position and the bone window size after configuring the projection edge as a mask.
4. The method of claim 3, wherein the clipping the brain vein model according to the bone window position and the bone window size comprises:
s206: by setting the edge of the mask after the equal ratio amplification as a clipping boundary, and then vector along the pathIs cut for the cerebral vein sub-model.
5. The method of brain surgical patient head presentation based on pose estimation according to claim 4, wherein said creating a pose estimation training dataset comprises:
s301: selecting a blood vessel peripheral point as a seed growth starting point according to the point set of the cut brain vein sub-model to construct a brain vein model, and presetting a brain vein bifurcation judging principle for judging whether bifurcation occurs or not based on the seed growth starting point;
s302: extracting an isoplane value of a brain vein blood vessel model through a marching cube algorithm to obtain a brain vein blood vessel surface network;
s303: judging the brain vein surface network according to the brain vein bifurcation judging principle, confirming bifurcation points of all branches of the brain vein, marking the bifurcation points and selecting the bifurcation points as key points;
S304: traversing each branch of the cerebral vein model until all branches corresponding to the bifurcation points are cut off, and storing the key points and the central points of the cerebral vein sub-model to a three-dimensional point set K 3d In (a) and (b);
s305: introducing a cerebral vein sub-model of the cerebral vein model containing the resected branches into a three-dimensional graphics processing engine to manufacture a data set;
s306: parallel projection of the cerebral vein sub-model and collection of the three-dimensional points K 3d Projecting into a two-dimensional RGB image to obtain the brain vein sub-model as a two-dimensional RGB image and a set K 3d Two-dimensional point set K of (2) 2d ；
S307: constructing a mask area of a two-dimensional RGB image and a three-dimensional point set, obtaining RGB images of brain vein sub-models under different visual angles by changing camera view point rendering, and recording pose matrixes P of the corresponding brain vein sub-models under all camera view points render ；
S308, adding a brain vein semantic excision tag, a brain vein two-dimensional point set and a pose and pose matrix P to the two-dimensional RGB image in the data set render A pose estimation training dataset is obtained.
6. The method of claim 5, wherein iteratively training the brain vein pose estimation network model via the pose estimation training dataset comprises:
S309: inputting the pose estimation training data set into a brain vein pose estimation network model, and outputting a class label of each pixel in the RGB image through a backbone network, wherein the class label comprises a unit vector field for judging whether the pixel belongs to a brain vein and expressing the direction of the pixel towards a key point;
s310: generating a set of assumed key points for all the key points, wherein the assumed key points are two pixels selected randomly, and the intersection point of pixel vectors is used as the assumed key point;
s311: observing each set of hypothesized key points in sequence by meeting the condition of highest confidence, and selecting the highest scoring point in each set of hypothesized key points as a target key point of the brain vein pose estimation network model;
s312: and solving the pose P of the brain vein in the RGB image input into the brain vein pose estimation network model through an EPNP algorithm based on a preset three-dimensional key point and a target key point.
7. The brain surgery patient head presentation method based on pose estimation according to claim 1, wherein the calculation formula for obtaining the brain vein pose of the patient by inputting the RGB image into the brain vein pose estimation network model is:
N all ＝N pixel-1 +N pixel-2 ……+N pixel-n ，
After the images are sequentially input into the trained network model, each image can obtain a pixel and pose matrix P belonging to the cerebral veins, and each image is segmented according to the time sequence of the input images and the semantics of the imagesUpdating the actual cerebral vein pose P of a patient once word Wherein N is pixel-1 、N pixel-2 、......、N pixel-n Represented between 0 and 0.The number of pixels belonging to the cerebral vein in the nth image inputted within 5 s.
8. A brain surgical patient head presentation system based on pose estimation, the system comprising:
the preoperative model reconstruction module is used for acquiring preoperative head two-dimensional image data of a patient, reconstructing the two-dimensional image data into a brain model of the patient based on three-dimensional reconstruction, wherein the brain model comprises a skin submodel, a skull submodel, a brain tissue submodel, a brain artery submodel, a brain vein submodel and a focus submodel;
the preoperative model processing module is used for pre-cutting the cerebral vein sub-model through the skull sub-model by adopting Boolean operation, determining the bone window position and the bone window size of the brain model after selecting a target operation path, and finishing the cutting processing of the cerebral vein model according to the bone window position and the bone window size;
The preoperative model training module is used for creating a brain vein pose estimation network model and a pose estimation training data set, and iteratively training the brain vein pose estimation network model through the pose estimation training data set;
the head presenting model of the patient in operation is used for acquiring RGB images of the craniotomy area of the patient in operation, inputting the RGB images into the brain vein pose estimating network model to acquire the brain vein pose of the patient, fusing the brain vein pose with the head pose of the patient to carry out virtual reality presentation, and controlling real-time update of the virtual reality presentation through preset gestures.
9. A brain surgical patient head presentation device based on pose estimation, comprising:
at least one processor, and,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the pose estimation-based brain surgical patient head presentation method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing computer-executable instructions for causing a computer to perform the pose estimation-based brain surgical patient head presentation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311528057.4A CN117557724A (en) | 2023-11-15 | 2023-11-15 | Head presentation method and system for brain surgery patient based on pose estimation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311528057.4A CN117557724A (en) | 2023-11-15 | 2023-11-15 | Head presentation method and system for brain surgery patient based on pose estimation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117557724A true CN117557724A (en) | 2024-02-13 |
Family
ID=89814173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311528057.4A Pending CN117557724A (en) | 2023-11-15 | 2023-11-15 | Head presentation method and system for brain surgery patient based on pose estimation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117557724A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021121126A1 (en) * | 2020-07-28 | 2021-06-24 | 平安科技（深圳）有限公司 | Deep learning-based brain tumor image division method, device, apparatus, and medium |
| CN113744272A (en) * | 2021-11-08 | 2021-12-03 | 四川大学 | Automatic cerebral artery delineation method based on deep neural network |
| CN114419181A (en) * | 2022-02-08 | 2022-04-29 | 东软医疗系统股份有限公司 | CTA image reconstruction method and device, display method and device |
| JP7157988B1 (en) * | 2022-02-15 | 2022-10-21 | 国立大学法人 東京医科歯科大学 | human head model |
| CN116420107A (en) * | 2020-11-09 | 2023-07-11 | 皇家飞利浦有限公司 | Deep brain path planning using augmented reality |
| US20230252627A1 (en) * | 2021-12-22 | 2023-08-10 | RealizeMD Ltd. | Systems and methods for generating clinically relevant images that preserve physical attributes of humans while protecting personal identity |
-
2023
- 2023-11-15 CN CN202311528057.4A patent/CN117557724A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021121126A1 (en) * | 2020-07-28 | 2021-06-24 | 平安科技（深圳）有限公司 | Deep learning-based brain tumor image division method, device, apparatus, and medium |
| CN116420107A (en) * | 2020-11-09 | 2023-07-11 | 皇家飞利浦有限公司 | Deep brain path planning using augmented reality |
| CN113744272A (en) * | 2021-11-08 | 2021-12-03 | 四川大学 | Automatic cerebral artery delineation method based on deep neural network |
| US20230252627A1 (en) * | 2021-12-22 | 2023-08-10 | RealizeMD Ltd. | Systems and methods for generating clinically relevant images that preserve physical attributes of humans while protecting personal identity |
| CN114419181A (en) * | 2022-02-08 | 2022-04-29 | 东软医疗系统股份有限公司 | CTA image reconstruction method and device, display method and device |
| JP7157988B1 (en) * | 2022-02-15 | 2022-10-21 | 国立大学法人 東京医科歯科大学 | human head model |
Non-Patent Citations (3)
| Title |
|---|
| SAQLAIN RAZZAQ; NIMRA MUBEEN; URWAH KIRAN; MUHAMMAD ADEEL ASGHAR; FAWAD FAWAD: "Brain Tumor Detection From MRI Images Using Bag Of Features And Deep Neural Network", IEEE, 26 November 2020 (2020-11-26) * |
| 王子罡, 唐泽圣, 王田苗, 刘达, 吕洪波, 申皓: "基于虚拟现实的计算机辅助立体定向神经外科手术系统", 计算机学报, no. 09, 12 September 2000 (2000-09-12) * |
| 陈光忠，李铁林，李昭杰，舒航，曾少建: "脑动静脉畸形病巢周围毛细血管组织形态与体视学测量", 中国神经精神疾病杂志, 30 June 2007 (2007-06-30) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10951872B2 (en) | Augmented reality guidance for spinal procedures using stereoscopic optical see-through head mounted displays with real time visualization of tracked instruments | |
| US10143522B2 (en) | Systems and methods for planning hair transplantation | |
| US10357316B2 (en) | Systems and methods for planning hair transplantation | |
| US11961193B2 (en) | Method for controlling a display, computer program and mixed reality display device | |
| CN107016685A (en) | A kind of surgical scene augmented reality projective techniques of real-time matching | |
| US20230248439A1 (en) | Method for generating surgical simulation information and program | |
| CN110547869B (en) | Preoperative auxiliary planning device based on virtual reality | |
| CN116196098B (en) | Method, system, medium and electronic equipment for generating planting planning scheme | |
| KR102433473B1 (en) | Method, apparatus and computer program for providing augmented reality based medical information of patient | |
| KR101401338B1 (en) | Surgery simulation method using virtual knife and apparatus thereof | |
| CN117557724A (en) | Head presentation method and system for brain surgery patient based on pose estimation | |
| TW202207242A (en) | System and method for augmented reality spine surgery | |
| CN114270408A (en) | Method for controlling a display, computer program and mixed reality display device | |
| CN114334096A (en) | Intraoperative auxiliary display method and device based on medical image and storage medium | |
| CN116385513A (en) | PSO-softPOSIT joint algorithm-based laparoscopic liver surgery navigation 2D/3D registration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |